WO2016079402A1

WO2016079402A1 - Data logging system and method

Info

Publication number: WO2016079402A1
Application number: PCT/FR2015/053078
Authority: WO
Inventors: Samia BENRACHI; Jean-Charles Masse; Pascal CONRATH
Original assignee: Bull Sas
Priority date: 2014-11-18
Filing date: 2015-11-16
Publication date: 2016-05-26
Also published as: FR3028636B1; FR3028636A1

Abstract

Data logging system (100) for structuring unstructured data and publication of at least one data structure associated with an article, said system comprising - a data acquisition module (10) configured to o receive unstructured data from a plurality of sources (1, 2, 3, 4, 5, 6) of heterogeneous data; o structure the set of said data into a pivot format; - a data processing module (20) configured to o record the structured data in a database; o index the recorded data; o generate at least one data structure from a plurality of recorded data in response to a request from a data management module (30), said data management module (30) being configured to publish at least said data structure.

Description

SYSTEM AND METHOD OF DATA JOURNALISM

The invention relates to a data journalism system for data management and the publication of data structures relating to such data.

In order to produce articles, journalists are first required to choose an information subject. Then, in order to build a narrative around this topic, they look for information from primary or secondary data sources. Over time, the Internet has become an essential tool for the journalist, thanks to the wealth of available data sources, such as articles, blogs, official sources of data (eg governments, communities, businesses), social networks, or even more generally open data. In addition, journalists frequently choose to accompany their articles with visual aids, for example images, graphics or computer graphics.

Among the different categories of journalism, data journalism, commonly referred to as data journalism, is a fast-growing form of journalism. Data journalism aims to exploit a set of encrypted and / or statistical data, to generate visual representations using graphics and / or computer graphics, often interactive. In this context, for example, an article published on a digital medium offers its readers the possibility to select data associated with a data structure (eg: graph, tables, tree) accompanying this article, and to manipulate this data. representation, allowing them to understand and memorize information more easily. Data journalism thus makes it possible to make the encrypted data talk by generating one or more data structures, and thus to transmit an informative content to the reader of an article, in a much more efficient manner than would the words of this article .

In a context of data journalism, journalists collect information from a wide variety of data sources (eg Internet sources), leading to the obtaining of data that is not usually structured, or otherwise structured in a very heterogeneous way. Such data, for example, may be presented in different electronic formats (text, tables, specific data structures). In order to generate a visual representation accompanying an article, it is therefore necessary initially to structure these data one by one. Such structuring work is tedious for the journalist and considerably lengthens the production time of an article. The realization of a visual representation, and more generally of a data structure, also requires various business skills that the journalist does not necessarily possess. For exemple :

- the choice of shapes and colors for the construction of a visual representation requires knowledge in visual semiotics or graphic semiology. It may therefore be necessary to call on a graphic designer;

performing an interactive visual representation may require the skills of an interaction designer.

It is therefore common, for a media company, to call on outside expertise, which incurs significant time and economic costs for the production of articles.

The invention aims to meet all the aforementioned drawbacks.

A first objective is to offer journalists, in the context of data journalism, a tool to reduce time or economic costs when producing an article.

A second objective is to propose a tool that meets the expectations of journalists, in the context of data journalism.

A third objective is to propose a tool facilitating the collection, structuring and use of data by journalists, in the context of data journalism.

A fourth objective is to propose a tool making it possible, from a heterogeneous and unstructured data set, to generate at least one data structure.

For this purpose, it is proposed, according to a first aspect, a data journalism system for structuring unstructured data and publishing at least one data structure associated with an article, this system comprising

a data acquisition module configured for: receiving unstructured data from a plurality of heterogeneous data sources;

o structure all of these data in a pivotal format;

a data processing module configured for:

o save the structured data in a database; o index the recorded data;

generating, from a plurality of stored data, at least one data structure in response to a request from a data management module, this data management module being configured to publish at least this data structure.

Advantageously, in this system, the data management module is configured to propose, via a graphical interface:

o tools for writing an article;

o the selection of one or more data indexed by the data processing module;

o a request to the data management module for the generation of a data structure from the selected indexed data;

o the publication of an article written with the requested data structure.

Advantageously, in this system:

each structured data item is associated with at least one keyword able to identify this datum;

the data processing module configured for:

o evaluate the relevance of each structured and indexed data from at least one predetermined metric;

o provide the data management module with a set of keywords associated with the data deemed relevant;

the data management module is configured to propose via a graphical interface the selection of one or more data indexed by the data processing module from the keywords communicated by the data processing module. Advantageously, in this system, a predetermined metric is selected from a list of metrics including: the number of occurrences of a keyword common to different data, the popularity or visibility of a subject or set of keywords on the Internet, the frequency of use over time of a keyword by different users of the data journalism system.

Advantageously, in this system, the data is formatted according to a pivot format in accordance with the IPTC standard.

Advantageously, in this system, the data processing module is configured to record the structured data in a NoSQL database.

Advantageously, in this system, the data management module is configured to propose when producing an article, the selection of one or more models of data layout, for structuring the article according to its publication support.

It is proposed, in a second aspect, a method for structuring unstructured data and publishing at least one data structure associated with a data journalism article, which method comprises:

a step of receiving unstructured data from a plurality of heterogeneous data sources;

a step of structuring all of these data in a pivot format;

a step of saving the structured data in a database;

a step of indexing the recorded data;

a generation step from a plurality of stored data of at least one data structure in response to a request;

a step of publishing the data structure with a data journalism article.

Advantageously, in this method:

each structured data item is associated with at least one keyword capable of identifying this datum;

the data structuring step is performed according to a pivot format in accordance with the IPTC standard;

this method further comprising the following steps:

a step of evaluating the relevance of each structured and indexed data item from at least one metric predetermined, this metric being selected from a list of metrics including the number of occurrences of a keyword common to different data, the popularity or visibility of a subject or set of keywords on the Internet, the frequency of use at the course of time of a keyword;

a step of communicating a set of keywords associated with the data deemed relevant;

a step of proposing the selection of one or more data indexed from the communicated keywords.

It is proposed, in a third aspect, a computer program product implemented on a memory medium, capable of being implemented within a computer processing unit and including instructions for the implementation of the summary method. above.

Other objects and advantages of the invention will become apparent from the description of the embodiments, given below with reference to the accompanying drawing, in which Fig. 1 illustrates a data journalism system according to one embodiment.

FIG. 1 shows a data journalism system 100 configured to manage a set of data from their retrieval in one or more formats until their publication according to one or more selectable format (s).

According to various embodiments, the data journalism system 100 comprises the following modules:

a module 10 for acquiring data;

a data processing module 20;

a data management module 30.

The data acquisition module 10 is configured to receive (unidirectional arrows 101, 102, 103, 104, 105, 106) a set of unstructured data from one or more sources 1, 2, 3, 4, 5, 6 of data. Advantageously, the data from sources 1, 2, 3, 4, 5, 6 of data are very heterogeneous natures and provenances. For exemple :

data sources 1, 2, 3 are data sources available on the Internet, respectively offering data recorded in XML, PDF, CSV formats. These data are at as examples, retrieved from their storage location via the data acquisition module 10, which is configured to perform this operation periodically or at specified times;

the source 4 of data is a relational database, accessible by the data acquisition module 10 from a public or private communication network, and proposing a set of heterogeneous data such as text and multimedia files. Such a database is, by way of example, accessed periodically by the data acquisition module 10 in order to update the contents proposed by the data journalism system 100;

the data source 5 corresponds to one or more files in Excel or text formats provided directly by a user to the data journalism system 100 via an appropriate graphical interface;

the data source 6 proposes a set of RSS data streams downloaded, for example, periodically by the data acquisition module 10.

According to various embodiments, the data associated with sources 1, 2, 3, 4, 5, 6 of data (heterogeneous) are extracted periodically by the data acquisition module 10. To do this, a task planning or scheduling tool can be associated with the data acquisition module 10.

The data acquisition module 10 is, moreover, configured to structure the set of data received, according to a predetermined pivot format, that is to say a format for structuring in a common manner all the data received. Advantageously, such a structuring is performed automatically by the data acquisition module 10. Thus, since a set of unstructured data received and heterogeneous formats, the data acquisition module 10 outputs (arrow 107) a set of data formatted in a similar structure, able to be directly used in read / write during the subsequent realization of any article and / or computer graphics by a journalist. Advantageously, the data acquisition module 10 is configured to structure the data (unstructured / heterogeneous) that it receives as input to a pivot format in accordance with the IPTC standard, acronym for "International Press Telecommunications Council". By way of example, the data acquisition module 10 formats the data it receives as input according to the news exchange format NewsML, the acronym for "News Markup Language", according to the format NewsML-G2. or in NITF format, the acronym for "News Industry Text Format," and then provides formatted output data. Advantageously, the formatting of the data received according to the IPTC standard makes it possible to obtain for each datum a structure comprising a set of metadata, the latter describing, by way of examples, numerical data and / or keywords associated with the datum. In another example, the data is formatted by the data acquisition module 10 so as to be compatible with any NRCS newsroom computer system, the acronym for "Newsroom Computer System". More generally, the data acquisition module 10 is configured to format the received data according to a predetermined pivot format, this format being chosen so as to allow the structuring of these data and their respective metadata, and allow their use for production. journalistic articles. According to one embodiment, the data acquisition module 10 is made according to the model of an ETL extracto-charger component, acronym for ". Extract Transform Load ", configured to perform the extraction and data formatting operations outlined above.

The data formatted according to a pivot format, and provided at the output of the data acquisition module 10, are then communicated to the input of the data processing module 20. Advantageously, the data processing module 20 then stores all the formatted data in a database. In one embodiment, the database used is a NoSQL type database associated with a database management system DBMS such as Cassandra. Advantageously, the database is made in such a way as to avoid any redundancy of data, while at the same time centralizing all of these data, which are structured according to the same pivot format. In one embodiment, the database is structured in columns by the data processing module 20, according to different keywords (tags), each data item of the database being associated with at least one keyword. A key word can as an example, refer to a description provided by the author of the data, or to an element characterizing the data allowing its identification (eg identification of a unit of measure). Such a key word may, by way of example, be extracted from the metadata of a data item received by the data acquisition module 10, when formatting it to a pivot format and identified by the data processing module 20 as one of the characteristic fields of the data.

The data present in the database of the data processing module 20 are then indexed according to their respective keywords and / or their content, via an indexing method implemented in this module or an appropriate tool integrated in this module. module, for example via "ElasticSearch". More specifically, the data processing module 20 is configured during indexing to associate one or more tags with the data stored in the database of this module. The data processing module 20 is therefore able to establish a correspondence between a keyword and one or more data stored in its data table.

Moreover, in order to value all the data recorded in its database, the data processing module 20 is configured to evaluate the relevance of each data item. In one embodiment, the evaluation of the relevance of a piece of data is, for example, carried out by assigning a score to this piece of data, as a function of one or more predefined metrics. These metrics are, as examples, chosen from a list of metrics including the number of occurrences of the same tag (keyword) common to different data, the popularity or visibility of a subject or set of keywords on the Internet, or the frequency of use over time of the same keyword by different users of the system 100de journalism data. If it is desired to implement this last metric, it then ensures to record in the data processing module 20, a history of use of the tags for the same user, or for a set of users of the system 100 of data journalism. Advantageously, all data recorded, indexed and evaluated in the database of the processing module 20 is therefore an enriched knowledge base thanks to the diversity and heterogeneity of its sources, while proposing a set of structured data. Such a base can, moreover, be directly enriched by a journalist, via the injection of primary data, such as an interview conducted by the journalist and directly provided by the latter to the system 100 of journalism data.

The data processing module 20 is moreover configured for

performing one or more data structures, such as graphical representations, from a plurality of data stored in its database. As examples, the data processing module 20 implements one or more methods capable of producing point curves, tables, histograms, circular diagrams, maps that can be interactive, comparative bubbles, interactive temporal representations. and / or relationship graphs. In particular, the data processing module 20 is configured to propose and construct interactive representations, allowing the user to select or deselect data, such representations allowing the user a dynamically adjustable reading depth; establish one or more correlations between data recorded in its database, and make comparisons between data of different types or sources. These data can, for example, be compared directly when they have similar units of measurement, or in the case of separate units be correlated via a statistical means, such as a linear regression calculation.

Advantageously, the data processing module 20 is configured to transmit (arrow 108) to the input of the data management module 30, a set of recorded data, one or more data structures (eg graphs), and / or one or more tags relating to the recorded data. This information can be transmitted to the data management module 30. in a thorough manner, for example via the suggestion of one or more tags associated with recorded data, and deemed relevant by the data processing module 20;

derived in response to a plurality of requests, for example NoSQL type, sent by the data management module 30 to the data processing module 20 (arrow 109).

Advantageously, the data management module 30 implements a graphical interface, accessible for example via a web browser, configured to:

- manage users' access to the 100 system of data journalism, as examples to allow several journalists to work on the same article, prioritize users), assign roles to them (eg administrator, registered user, guest, content access permissions (eg read / write) and data journalism system 100 tools;

allow a user to send data requests (eg of the NoSQL type) and infographics associated with keywords (tags), to the data processing module 20, for example via the use of a search engine. research. More particularly, the graphic interface is designed to offer to any user, the selection of one or more keywords (tags) and one or more graphic representations in a catalog, proposed by the data processing module 20 As examples, the user can send a request to obtain

a simple temporal representation of a data item recorded by the data processing module 20, for example the temporal evolution of the gross domestic product of a specific country;

a correlation between at least two data recorded by the data processing module, for example the comparison of the evolution of the gross domestic product with the human development index of a specific country over a period of time that can be selected via the 'graphic interface ;

an interactive graphical representation generated by the data processing module, offering a dynamically configurable display, via the possibility of selecting or deselect graphically the constituent data of this representation;

communicate to the user a set of keywords (tags) deemed relevant by the data processing module 20, for example a set of keywords referring to a current topic in high visibility on the Internet. As explained above, the relevance of a data item or a tag is evaluated by the data processing module 20 according to predefined metrics. Thus, a list of keywords, may for example be pushed by the data processing module 20, and suggested via the graphical interface of the user data management module 30 of the data journalism system 100 . For example, a keyword list is provided based on the keyword search history that a journalist has previously made, or based on the popularity of keywords by a set of users of the system. data journalism. Moreover, in order to refine the evaluation of relevance of the keywords, the graphical interface may optionally propose to the user, to assign a score of relevance to the returned data, this score being then communicated to the processing module 20. data. The data processing module 20 then reevaluates the relevance of the data according to the score assigned by the user and his own metrics;

- offer the user the selection, sorting, filtering and / or evaluation of one or more sources 1, 2, 3, 4, 5, 6 of data. Advantageously, such possibilities facilitate the decision-making by the user of the data journalism system 100, in the choice of the data to be used for the production of his article, and make it possible to continuously provide reliable and up-to-date sources;

provide tools for writing articles, and the integration of user requested infographics and returned by the data processing module 20 in the articles written; allow the injection (dashed arrow 110) of primary data into the data journalism system 100, i.e. data directly produced by the journalist. For example, the graphical interface optionally offers the journalist to add in the system 100 of data journalism an interview he has himself made. This is then communicated to the data acquisition module 10 for formatting and then transmitted to the data processing module 20 for storage and indexing;

proposing the selection of one or more models of data layout, that is, templates, in order to structure an article, according to the publication medium for which it is intended;

allow the publication (arrow 111) of articles to one or more computer media 7 for example to smartphones, tablets, or web browsers. Advantageously, the publication of an article can be performed dynamically, that is to say, be scalable over time. For example, a journalist who has already published an infographic relating to one or more data, can evolve this infographic by taking into account data newly added or updated in the database of the data processing module 20 . In another example, the journalist publishes a first draft article, which it embellishes over time with one or more graphic representations proposed by the system 100 of data journalism. The system 100 of data journalism thus makes it possible to ensure the management of the life cycle of the data, by updating them, updating their source, and putting them forward to the journalist for the production of articles;

allow the export (arrow 112) in a predetermined format 8, for example the printing of an article in a PDF format.

According to one embodiment, the data management module 30 is made on the model of a content management system CMS, acronym for "Content Management System", configured to perform the set of operations described above.

Advantageously, thanks to the joint use of the data acquisition module 10, the data processing module 20 and the data management module 30, the system 100 of journalism of data facilitates the work of journalists in many ways. Such a system allows, in particular, from a large number of sources that can be very heterogeneous and initially not necessarily containing structured data, to respond to requests from journalists via the return of structured information.

Advantageously, the data journalism system 100 collects a set of unstructured data from these heterogeneous sources and formats them automatically in order to obtain usable data for the realization of any data structure, such as a graphical representation. . Advantageously, the choice of an appropriate formatting makes it possible to make this data interoperable with any journalism system and more generally any information-based system, as examples the data set is compatible with the IPTC standard, the computer systems of NRCS newsroom, social networks or microblogging tools.

Such a system thus allows the constitution of a database of very rich nature, thanks to the wide variety of existing sources, such as Internet sources (sites, articles, blogs) and data injected by journalists into the system. Such a database is also scalable, that is to say, takes into account as well, the addition of new data, the updating of data as well as metrics associated with them, thus providing journalists with a Dynamic and reliable information.

Advantageously, the graphic interface proposed by the system

100 of journalism data, facilitates the decision-making of the journalist in the choice of a topic of article writing, by highlighting a set of keywords, referring to topics in high visibility on the Internet . Such a graphical interface also makes it possible to select and produce one or more data structures, such as graphical, interactive and configurable representations, out of a wide choice proposed by the data journalism system 100. The journalist can thus quickly determine a representation that he considers relevant for his article, and generate it automatically, without using a specialized computer graphics service. The journalism system 100 of data further allows the journalist to directly select one or several layouts depending on the media on which he wishes to publish his article, allowing the dissemination of articles, graphics and computer graphics products to a wider audience.

Advantageously, the data journalism system 100 therefore facilitates the work of the journalist and reduces the production costs of his articles, while ensuring interoperability with the production systems used in the media sector.

Claims

A data journalism system (100) for structuring unstructured data and publishing at least one data structure associated with an article, the system comprising

a data acquisition module (10) configured for

receiving unstructured data from a plurality of heterogeneous data sources (1, 2, 3, 4, 5, 6);

o structuring all of these data in a pivotal format that makes it possible to structure all the data received in a common manner;

a data processing module (20) configured for

o save the structured data in a database; o index the recorded data;

generating, from a plurality of stored data, at least one data structure in response to a request from a data management module (30), this data management module (30) being configured to publish at least this structure of data.

The system (100) of claim 1, wherein the module

(30) data management is configured to provide via a graphical interface

o tools for writing an article;

o selecting one or more data indexed by the data processing module (20);

a request to the data management module (30) for the generation of a data structure from the selected indexed data;

o the publication of an article written with the requested data structure.

3. System (100) according to claims 1 or 2, wherein each structured data is associated with at least one keyword able to identify this data, and wherein

the data processing module (20) configured for

o evaluate the relevance of each structured and indexed data from at least one predetermined metric; o communicating to the data management module (30) a set of keywords associated with the data deemed relevant; the data management module (30) is configured to propose, via a graphical interface, the selection of one or more data indexed by the data processing module (20) from the keywords communicated by the module (20) of data processing.

The system (100) of claim 3, wherein a predetermined metric is selected from a metric list comprising: the number of occurrences of a keyword common to different data, the popularity or visibility of a subject or set of keywords on the Internet, the frequency of use over time of a keyword by different users of the system (100) of data journalism.

5. System (100) according to any one of claims 1 to

4, wherein the data is formatted according to a pivot format in accordance with the IPTC standard.

The system (100) according to any one of claims 1 to

5, wherein the data processing module (20) is configured to store the structured data in a database

NoSQL.

7. System (100) according to any one of claims 1 to

6, in which the data management module (30) is configured to propose when producing an article, the selection of one or more data layout templates, making it possible to structure the article as a function of its publication support.

8. Method for structuring unstructured data and publishing at least one data structure associated with a data journalism article, which method comprises

a step of receiving unstructured data from a plurality of heterogeneous data sources (1, 2, 3, 4, 5, 6); a step of structuring all of these data in a pivot format for structuring in a common manner all the data received;

a step of recording the structured data in a database; a step of indexing the recorded data;

- a step of publishing the data structure with a data journalism article.

The method of claim 8, wherein

this method further comprising the following steps

a step of evaluating the relevance of each structured and indexed data item from at least one predetermined metric, this metric being chosen from a list of metrics comprising the number of occurrences of a keyword common to different data, the popularity or visibility of a topic or set of keywords on the Internet, the frequency of use over time of a keyword;

10. Computer program product implemented on a memory medium, capable of being implemented within a computer processing unit and comprising instructions for the implementation of a method according to one of claims 8 or 9.