WO2016085414A1

WO2016085414A1 - Method to lower decline in watching channels during commercial breaks and a connection

Info

Publication number: WO2016085414A1
Application number: PCT/SK2015/000005
Authority: WO
Inventors: Lujza BUBÁNOVÁ; Martin FLOREK; Martin KRAVEC; Boris ŠEBOŠIK
Original assignee: JOHN SMITH s.r.o.
Priority date: 2014-11-27
Filing date: 2015-11-27
Publication date: 2016-06-02

Abstract

I n the method to lower decline in watching channels during commercial breaks, in one transmission direction from the audio sample from TV scanned by a mobile platform the currently selected channel is recognized and in the second transmission direction, locked special content to the television advertising enters the mobile platform. Opening special content to television advertising on a mobile platform is available at the beginning of the commercial break while watching the same selected channel. A system connection to lower decline in watching channels during commercial breaks consists of a sampling server (1) with input (2) from sources of broadcasting TV signals, further it consists of a content server (3), which is by a wire-less bi-directional connection connected with a mobile platform (4) with a control application. There is also a recognition server (5) there which is connected with sampling server (1) and also with content server (3). The content server (3) contains the first information block (6) with special content to television advertising and also the second information block (7) with aggregated relevant contents on social networks.

Description

Method to lower decline in watching channels d u ring commercial breaks and a connection

Technical Field of the Invention

The invention relates to the method to lower decline in watch ing channels du ring commercial breaks and a connection to this method . The invention falls within the telecommunication and advertising technology.

Background of the Invention

In the current state of the art there are standard advertising systems to display advertising shots or banners in electronic equipment as TV. Advertising as such is paid by the advertising submitter and thereby there is no response on watching of advertising until its end . Not watching advertising shot until its end significantly lowers the effect of advertising at the advertising visitor. Furthermore, one of the biggest problems and sources of economic losses for the TV broadcasters is decline in watchi ng their channels during commercial breaks. The viewers use to switch to another channel d uring them, leave the screen and stop watch ing events on TV or at this time they are not with in the reach of television sets. The stated negative facts led to the effort to create a concept of such advertising system for TV which would ensu re preservation of viewing their channels during commercial breaks, whereby there would be at the same time control of watch ing advertising shots for the advertising submitter. This concept would require particularly recognition of the watched TV program whose sound would be scanned by a mobile phone. A few services are dedicated to the recognition of sound recorded by a mobile phone. The most important ones are Shazam , Soundhound and Mufin . The service Shazam described in the patent US 6 , 990 ,453 is the most widespread and works on the basis of a comparison of so-called d igital fingerprints made of an aud io sample. All available services work on the basis of a comparison of fingerprints. Shazam analyzes the sound in which it finds significant features which are not affected even by strong noise and other various damages to the signal such as lossy audio compression of MP3 type or a poor-quality recording hardware. Such features are e.g. loud tones. Upon detection of these features also advanced methods of processing audio data such as cepstral analysis are used . A comparison of these features would be very slow and therefore they are not compared directly, but hashes are generated from these features. These hashes are generated from combinations of features (audio frequencies on which they are situated) and their distances from each other in time. In this manner manifold more hashes than features alone are generated, whereby the situations from damaged signals are well treated. Equal samples are used also to create a database of songs in which samples recorded by a mobile device are then searched. A comparison of hashes is very fast and therefore they are able to search in a database with millions of songs created in advance. Currently, Shazam recognizes also some selected TV shows.

Soundhoud described in published patent application US 2013/ 0044885 is the second most widely spread music recognition service. Similarly as Shazam they are based on digital fingerprints generated from the recorded audio sample. These fingerprints are generated from robust features which are not affected by noise or other damages of the signal. Unlike with Shazam, no hashes are generated from these features, but matrices which capture a relationship between the most striking feature and the second and third most striking feature in the recorded sample. These matrices are then continuously sent to the server which compares them with matrices in a database of samples of songs created in advance. The songs in the database have also more than one matrix assigned to them which describes them.

Mufin is a service which recognizes the live broadcast on selected television channels. In addition to recognition of a show and TV channel they can recognize also advertising. They use their own d igital audio imprint "Audiol D" and try to find and encode features in the sound which are not affected by various damages. Furthermore, in the published patent applications US2014/0188592 and US2014/0012572 addition of extra data to the recognized sound is described, e.g. of relevant advertising.

Furthermore, a company Blue Spike is known which e.g. on the basis of a patent US 7,346,472 creates an abstract from the music sample which it compares then with a database of abstracts with a result which song it is. The method may be software-implemented generally to digital data and is not limited to sound data.

In the state of the art a system Tune Hunter is known in which samples of recorded sound from a portable device (mobile phone, keychain, pager, etc.) are scanned which are then compared with a database of music samples created in advance. The system offers even relevant items for sale to the recognized sound, such as the song itself. The main difference towards other approaches is in sending and comparing the individual audio samples and not their simplification as hash or matrix as described e.g. in the patent US 6,941 ,275.

Current methods for recognizing songs work on the detection of features in audio samples which can survive noise and various damages to signals such as MP3 lossy compression, poor-quality recording hardware, and so on. These samples are then encoded in another form (hash, matrix, etc.) which captures a relationship between these features and can be easily compared. Besides Mufin, most systems compare a sample recorded by a mobile device with a database created in advance.

Based on the above stated shortcomings of the recognition of sound efforts have been made, resulting in the below described invention, which solves a method to lower decline in watching channels during commercial breaks and a system connection. Summary of the Invention

The above stated shortcomings are eliminated by a method to lower decline in watching channels during commercial breaks and a connection according to this invention characterized in that by the activation of the control application on a mobile platform in one transmission direction the audio sample of the currently watched program on TV scanned by a mobile platform is transmitted to the content server. The broadcast TV signal is sampled in a sampling server and proceeds to a recogn ition server. Also the already recognized currently watched selected channel enters from the recognition server to the content server. In the second transmission direction, locked special content to the television advertising enters the mobile platform . Opening special content to television advertising on a mobile platform is available at the beginning of the commercial break while watching the same selected channel. Advantageously at the beginning of a commercial break an alert to the current moment of broadcasting of advertising spot foregoes together with performance metrics of the given spot. Advantageously, in the second transmission direction aggregated relevant contents on social networks enter mobile platform in addition to sending special content to the television advertising .

The system connection to lower decline in watching channels during commercial breaks is based on the above stated method and consists of a sampling server with input from sources of broadcasting TV signals. Furthermore, it consists of a content server which is by a wireless bidirectional connection connected with a mobile platform with a control application . The mobile platforms of application users are smartphones, notebooks, laptops, tablets and similar devices activated in a data network such as mobile internet or in a mobile network, e .g . GSM . There is a recognition server which is connected with sampling server and also with content server. The content server contains at least the first information block with special content to television advertising and advantageously contains also the second information block with aggregated relevant contents on social networks which fill the content database. The content server contains also a request server and has an input for a generator of external signals of TV broadcasters. The sampling server contains a generator of descriptors of the source data. The recognition server contains e.g . an indexing server and a comparator with a database of ind ices.

The functionality of the connection is as follows. The basis of the recognition algorithm is input data. From an external device - a mobile phone, mobile platform , the sample is used for the recognition and the data from the sampling server is used as a reference sample in which a sample from an external device is searched . From both types of devices the sample is processed in the following manner. Gross input data from the data stream is processed by STFT (Short-time Fourier transform) algorithm . This processing creates a matrix of normalized data energies. From a matrix processed in such way the key points are extracted by means of a grid specified in advance. The coord inates of these points together with the processed matrix are input into BRI EF algorithm which creates for each key point a set of descriptors which describe the given key point and its surroundings. The descriptive set of descriptors from an external device is sent to the content server. The descriptive set of descriptors from sampling server is sent directly to the recognition server as a reference sample.

The sampling server processes m ultiple broadcasts from various sources. Each broadcast has an unambiguous identifier (I D of the channel). Each sample of input data from each broadcast is independently processed by the above described algorithm and together with the identifier sent to the recognition server as a reference descriptive sample of descriptors. The increased cred ibility of the results in the external device is achieved by sending multiple, various descriptive sets of descriptors for recogn ition and by increasing the number of identical answers of recognition .

The recogn ition server receives a descriptive set of descriptors from the content and sampling server. For each received descriptive set of descriptors from the sampling server the extractor extracts the precise time interval and creates an extracted descriptive set of descriptors. Accord ing to the identifier it assigns the created extracted descriptive set of descriptors to the agg regation of previous extracted descriptive sets of descriptors. I n reg ular intervals it extracts extracted descriptive sets of descriptors from aggregations and agg regates them to larger descriptive sets of descriptors representing overlapping longer time interval for the data stream accord ing to the identifier. Subsequently, it aggregates them in a time descriptive set of descriptors of the sample containing longer time intervals of the descriptive set of descriptors for multiple identifiers. The time sample of the descriptive set of descriptors is input to FLAN N algorithm , the result of which is the FLAN N index stored in a database. A key part is the creation of time samples of descriptive sets of descriptors of different lengths, which give rise to FLAN N indices of different lengths. It ensures that the amounts and lengths of searches necessary to recognize a sample are minimized . The recognition server keeps a minimum set of FLANN indices of d ifferent lengths in the database to ensure coverage. The incoming descriptive set of descriptors from the content server is sequentially compared in the comparator by LSH algorithm only with necessary FLAN N indices from the database in order to m inimize the search time. The LSH assigns to key points of the descriptive set of descriptors from the content server the closest key points of the just compared time sample of the descriptive set of descriptors. Subsequently, the comparison is assessed by a filtering algorithm , the score is evaluated and accord ing to the result either further search is triggered or the response is sent back to the content server.

The content server is a server containing information about broadcasts, information about working recognition servers and it is receiving external signals. Its primary task is to provide relevant and timely, supplementary information for the watched content. The content server maintains a list of currently available recognition servers. After receiving of a descriptive set of descriptors from an external source the available recogn ition server extracts and sends a descriptive set of descriptors for recognition . If it receives information that the recognition server is busy, it selects another recognition server. According to the response from the recognition server it subsequently creates corresponding content and sends it back to the mobile phone, mobile platform. It ensures thus automatic scaling in case of large amounts of requests for recognition . The content server is automatically scaled by third-party systems. Another task of the content server is after receiving of an external signal (trigger on TV - signal generator) to change, modify and alert external devices about a change of content.

For the purposes of this invention a definition of some terms is explained hereinafter:

Input data is a gross set of data from a source, e.g . data from an audio stream.

Sampling server is a device able to process broadcasts/data streams in real time.

Mobile platform is an external device, a mobile phone, a device capable of processing the recorded sound.

Matrix of normalized energies is an output matrix from Short-time Fourier transform , for audio input data the x-axis are overlapping time sections and y-axis frequency intervals.

BRIEF is an algorithm of creating a matrix of normalized energies of the input data.

Key point is coordinates of an extracted point from a matrix of normalized energies.

Set of descriptors is a description of a key point and its surroundings in a matrix of normalized energies.

Descriptive set of descriptors is a set of descriptors for each key point in a matrix of normalized energies .

Content server is a device which operates automatic scaling , receives a descriptive set of descriptors from external devices and sends them to recognition to recognition server, receives signals from external sources. FLAN N index is a structure for fast search of samples.

LSH is an algorithm for fast search of the descriptive set of descriptors in FLANN index.

Time sample of the descriptive set of descriptors is a set of descriptive sets of descriptors from multiple broadcasts aggregated according to the identifier with an exact length .

Recognition server is a device that receives a descriptive set of descriptors from the sampling server, aggregates them in a time sample of a descriptive set of descriptors and creates a database of FLANN indices. Subsequently, after receiving of the descriptive set of descriptors from the content server it compares it only with the necessary set of FLAN N ind ices and sends back the response.

The advantages of the method to lower decline in watching channels during commercial breaks and the system connection according to this invention consist in the fact that the system will offer viewers continuous experience when watch ing television by bridg ing commercial break in the form of a special content. It reduces their frustration of interruption of fun and at the same reduces the motivation to switch the channel. This content is locked during usual broadcast of shows. It is un locked at the beginn ing of a commercial break and is available on ly if the viewer does not switch to another channel, since the application recognizes the currently selected channel. The mechanism of un locking special content at the time of each commercial break works on a basis of processing of signal on its beginning and end. The system also serves as an aggregator of relevant content on social networks. It recognizes the individual moments in broadcasting , including advertising broadcasting. It enables thus bringing extended information on the television advertising to the application . The system continuously monitors the broadcast and thus recognizes the broadcast advertising and alerts its submitter to the current moment of broadcasting of advertising spot together with respective performance metrics of the given spot. A connection to an interface of visual graphic generator of broadcasters allows direct output of processed data from the system during the broadcast. Overview of Figures in the Drawings

A method to lower decline in watching channels d u ring commercial breaks and a system connection accord ing to the invention explained in the drawings, where Fig. 1 is a fundamental scheme of the system connection . Fig. 2 depicts a more detailed scheme of the system connection. Fig . 2 depicts the scheme of signal sampling .

Embodiment Examples

It is understood that individual invention embodiments are presented for illustration and not as limitations to technical solutions. Technology experts will find or be able to discover many equivalents to the specific invention embodiments through no more than routine experimenting. Such equivalents will also fall within the scope of the following patent claims. Technology experts can't have a problem with the optimum system design ; therefore these features aren't resolved in detail.

Example 1

I n this example of concrete invention embodiment a method to lower decline in watch ing channels during commercial breaks is described . It is based on the fact that by the activation of the control application on a mobile platform in one d irection in one transmission direction the audio sample of the currently watched program on TV scanned by a mobile platform is transmitted to the content server. The broadcast TV signal is sampled in a sampling server and proceeds to a recognition server. Also the already recognized cu rrently watched selected channel enters from the recogn ition server to the content server. In the second transmission direction, locked special content to the television advertising and aggregated relevant contents on social networks in add ition to send ing special content to the television advertising enter the mobile platform . Opening special content to television advertising on a mobile platform is available at the beginning of the commercial break while watching the same selected channel. At the beginning of a commercial break an alert to the current moment of broadcasting of advertising spot foregoes together with performance metrics of the given spot.

Example 2

In this example of concrete invention embod iment the system connection to lower decline in watching channels d uring commercial breaks fundamentally shown in Fig . 1 and in more detailed connection shown in Fig. 2 is described . It consists of the sampling server 1 with input 2 from the distributors 20 of sources of broadcasting TV signals. It consists of the content server 3 which by a wire-less bi-d irectional connection connected with a mobile platform 4 by smartphone with a control application . I n the connection there is recognition server 5 integrated which is connected with sampling server 1 and also with content server 3. The content server 3 contains the first information block 6 with special content to television advertising and contains also the second information block 7 with aggregated relevant contents on social networks fill the content database 13. The content server 3 contains also a request server 9 has an input 8 for a generator 14. of external signals of TV broadcasters 15. The content server 3 further contains a database |7_ of metadata of TV channels, block 1_9 of TV time schedules and a time block 18. The sampling server 1 contains a generator 10 of descriptors of the source data. The recognition server 5 contains e.g . an indexing server H and a comparator 2. with a database of indices 16. The mobile platform 4 contains a mod ule 2J_ of a TV set, generator 22 of descriptors of the compared data, generator 23 of requests, the display module 24_ of the relevant content and finally the time module 25. Example 3

In th is example of concrete invention embodiment the actual sampling and recognition of the input data from the data streams as shown in Fig. 3 is described . Gross input data from the data stream 26 is processed by STFT 36 (Short-time Fourier transform) algorithm . This processing creates a matrix 2_7 of normalized data energies. From a matrix 27 processed in such way the key points are extracted by means of a grid specified in advance. The coordinates of these points together with the processed matrix are input into BRI EF algorithm, generator 28 which creates for each key point a set of descriptors which describe the given key point and its surroundings. The descriptive set of descriptors 29 from an external device, mobile platform 4 is sent to the content server 3. The descriptive set of descriptors 29 from the sampling server 1 is sent directly to the recognition server 5 as a reference sample. The recognition server 5 receives a descriptive set of descriptors 29 from the content and sampling server 3, 1_. From each received descriptive set of descriptors 29 from the sampling server 1 the extractor 30 extracts the precise time interval and creates an extracted descriptive set of descriptors 32. According to the identifier 3J_ it assigns the created extracted descriptive set of descriptors 32 to the aggregation of previous extracted descriptive sets of descriptors 35. In regular intervals it extracts extracted descriptive sets of descriptors 35 from aggregations and aggregates them to larger descriptive sets of descriptors representing overlapping longer time interval for the data stream according to the identifier 3J_. Subsequently, it aggregates them in a time descriptive set of descriptors of the sample containing longer time intervals of the descriptive set of descriptors for multiple identifiers. The time sample of the descriptive set of descriptors 33 is input to FLANN algorithm , the result of which is the FLAN N index 34 stored in a database. A key part is the creation of time samples of descriptive sets of descriptors 33 of different lengths which give rise to FLAN N indices 34 of different lengths. It ensures that the amounts and lengths of searches necessary to recognize a sample are minimized . The recognition server 5 keeps a minimum set of FLANN indices 34 of different lengths in the database to ensure coverage. The incoming descriptive set of descriptors from the content server is sequentially compared in the comparator 2 by LSH algorithm only with necessary FLANN indices 34 from the database in order to minimize the search time. The LSH assigns to key points of the descriptive set of descriptors from the content server 3 the closest key points of the just compared time sample of the descriptive set of descriptors 33. Subsequently, the comparison is assessed by a filtering algorithm, the score is evaluated and according to the result either further search is triggered or the response is sent back to the content server.

Industrial Applicability

Method of operation of interactive and efficient electronic advertising system and a system device according to the invention find utility in the communication technology and advertising.

Claims

C L A I M S

1 . Method to lower decline in watching channels d uring commercial breaks, characterized in that in one transmission direction from the audio sample from TV scanned by a mobile platform the currently selected channel is recognized and in the second transmission direction , locked special content to the television advertising enters the mobile platform; opening special content to television advertising on a mobile platform is available at the beginning of the commercial break while watching the same selected channel.

2. Method to lower decline in watching channels during commercial breaks according to Claim 1 , characterized in that at the beginning of a commercial break an alert to the current moment of broadcasting of advertising spot foregoes together with performance metrics of the given spot.

3. Method to lower decline in watching channels d uring commercial breaks accord ing to Claim 1 , characterized in that in the second transmission d irection agg regated relevant contents on social networks enter mobile platform .

4. System connection to lower decline in watching channels during commercial breaks, characterized in that it consists of a sampling server ( 1 ) with input (2) from sources of broadcasting TV signals, furthermore, it consists of a content server (3) , which is by a wireless bi-d irectional connection connected with a mobile platform (4) with a control application , whereby there is a recogn ition server (5) which is connected with sampling server (1 ) and also with content server (3); the content server contains (3) contains the first information block (6) with special content to television advertising .

5. System connection to lower decline in watching channels during commercial breaks according to Claim 4, characterized in that the content server (3) contains the second information block (7) with aggregated relevant contents on social networks.

6. System connection to lower decline in watching channels during commercial breaks according to Claim 4, characterized in that the content server (3) contains a request server (9) has an input (8) for a generator of external signals.

7. System connection to lower decline in watching channels during commercial breaks according to Claim 4, characterized in that the sampling server (1 ) contains a generator (10) of descriptors of the source data.

8. System connection to lower decline in watching channels during commercial breaks according to Claim 4, characterized in that the recognition server (5) contains an indexing server (1 1 ) and a comparator (12).