WO2015010555A1

WO2015010555A1 - Data blocking method and device

Info

Publication number: WO2015010555A1
Application number: PCT/CN2014/082237
Authority: WO
Inventors: 吴俊�; 张亮; 郭凯
Original assignee: 华为技术有限公司
Priority date: 2013-07-23
Filing date: 2014-07-15
Publication date: 2015-01-29
Also published as: CN104348571A; CN104348571B

Abstract

A data blocking method and device. The method comprises: using fingerprint values of a sliding window to respectively conduct a modulus operation on the first modulus value to the Nth modulus value, if the value is zero after conducting a modulus operation on one modulus value, taking data in an original data stream from the end position of a previous data block of the nth layer to the end position of the sliding window as current data blocks; when fingerprint values of the current data blocks are not stored in a fingerprint dictionary, if n is greater than 1, writing the current data blocks into a space occupied by a new data stream, so as to cover all the (n-1)^th layer of data blocks comprised in the current data blocks; otherwise, writing the current data blocks into the new data stream in sequence; otherwise, writing the fingerprint values of the current data blocks into the new data stream, so as to cover the fingerprint values of all the (n-1)^th layer of data blocks comprised in the current data blocks in the new data stream; otherwise, writing the fingerprint values of the current data blocks into the space occupied by the new data stream in sequence.

Description

Data blocking method and device

The present invention relates to communications technologies, and in particular, to a data blocking method and apparatus. Background technique

Repeating the transmission of the same or similar data wastes network resources. Redundant traffic removal

(Redundant Traffic Elimination, RTE for short) technology, which enables protocol-independent redundancy elimination and makes data communication more efficient. The existing RTE technology usually adopts a modular method based on Modular Exponential (MODP) to block the data stream according to the content of the file. The M0DP-based blocking method is implemented by a sliding window, for a given modulus value p (for example, 10010), when the fingerprint value of the sliding window (for example, 1111 0111 0010) is modulo the modulus value p 0, the data content between the end of the sliding window and the end of the previous data block is divided into a data block. When the fingerprint value of the sliding window (for example, 1111 0111 0010) is not modulo the modulus value p, the sliding window moves backward by one byte or a custom length. After obtaining the data block, the fingerprint value of the block is calculated, and compared with the fingerprint value of the stored data block in the fingerprint dictionary. If the same fingerprint value is detected, indicating that the redundant data content is found, the new The data stream is replaced with an identifier in the data stream to achieve the purpose of compressing traffic. If the same fingerprint value is not detected, the data chunk and its fingerprint are stored for redundant detection of subsequent chunking. The length of the identifier used to replace the data block is fixed. If the corresponding data block length is larger, the bandwidth saved during transmission is larger. However, large data partitioning means that the redundancy is coarse-grained, and the same data partitioning is difficult to repeat, resulting in a low redundancy rate.

At present, a hierarchical blocking scheme is proposed, which can achieve coarse-grained and fine-grained redundancy elimination at the same time to improve the deduplication ratio. The transmitting end uses the M0DP-based blocking method to block the original data stream, and obtains the data blocking sequence 101: the blocks SA, SB, SC, SD, SE, and SF, and replaces them with the identifiers of the corresponding blocks, for example, The block SA is replaced with an identifier R, resulting in an identifier sequence 102 (referred to as a layer 1 identifier): R , R , R , R and . The identifier sequence 102 is regarded as a general data stream, and the identifier sequence 102 is again partitioned using the MODP-based blocking method, and two data points of the identifier data are obtained. The block is also replaced by a 2-layer identifier sequence 103: R ² nR ² ₂ . For example, the content is the replacement of the identifiers R and , and the content of the layer 2 identifier is the identifier! ^And! ^ Replace with the 2-layer identifier R ² ₂ . From the data content, the layer 2 identifier represents the data blocks SA and SB, and R ² ₂ represents the blocks SC, SD, SE and SF. In the new data stream sent by the sender to the receiver, the data partition is replaced with the upper layer identifier as much as possible to save bandwidth.

However, in the above hierarchical blocking scheme, the layer 2 identifier is generated after the layer 1 identifier is segmented, and an indicator bit must be added to the layer 2 identifier to indicate that the identifier is a block. The identifier, or the identifier of the identifier. Otherwise, the receiving end cannot restore the original data. Therefore, the above hierarchical blocking scheme increases the overhead of the identifier and reduces the deduplication ratio. SUMMARY OF THE INVENTION Embodiments of the present invention provide a data blocking method and apparatus for improving data deblocking rate. In a first aspect, an embodiment of the present invention provides a data blocking method, including:

11. Obtain a fingerprint value of a sliding window on the original data stream, where an initial starting position of the sliding window is the same as a starting position of the original data stream, and a length of the sliding window is a preset length;

12. Using the fingerprint value of the sliding window, respectively modulating the first modulo value to the Nth modulo value, and for each modulo value, if the value after the value is zero, steps 13 and 14 are performed, The i-th modulo value is extended to the left by a preset number of bits to obtain an i+1th modulo value, where i is greater than or equal to 1 and less than any natural number of the N, and the N is a total of modulo values. Number

13. Data in the original data stream from an end position of a previous data block of the nth layer to an end position of the sliding window as a current data block, where the current data block is an nth layer data block Wherein n is one or more natural numbers that are greater than or equal to 1 and less than or equal to:

When the fingerprint value of the current data block is not stored in the fingerprint dictionary, storing the current data block into a block dictionary, and the fingerprint value of the current data block and the current data block in the block dictionary The index information in the fingerprint dictionary is stored in the fingerprint dictionary. When the n is greater than 1, the current data block is written in a space occupied by the new data stream to overwrite the current data block in the new data stream. All n-1th data blocks included, when the n is equal to 1, the current data block is sequentially written into the space occupied by the new data stream;

When the fingerprint value of the current data block is stored in the fingerprint dictionary and when n is greater than 1, the fingerprint value of the current data block is written in a space occupied by the new data stream to cover the new Data a fingerprint value of all n-1th data blocks included in the current data block in the stream, when the fingerprint value of the current data block is stored in the fingerprint dictionary, and when n is equal to 1, the current The fingerprint value of the data block is sequentially written into the space occupied by the new data stream;

14. Slide the sliding window on the original data stream toward the end position by the preset length, obtain the fingerprint value of the sliding window on the original data stream, and repeat steps 12-14 until the sliding window starts. The start position slides to the end position of the original data stream.

With reference to the first aspect, in a first possible implementation, the current data block is stored in a block dictionary, and a fingerprint value of the current data block and the current data block are in the block dictionary The index information is stored in the fingerprint dictionary, including:

When the current data block includes the data block of the n-1th layer, storing the fingerprint value of the current data block and the index information of the current data block in the block dictionary into the fingerprint dictionary, The index information of the current data block in the block dictionary is determined according to index information of a data block of the n-1th layer included in the current data block;

When the current data block does not include the data block of the n-1th layer, storing the current data block into the block dictionary, and the fingerprint value of the current data block and the current data block are in the The index information in the block dictionary is stored in the fingerprint dictionary.

With reference to the first aspect, or in combination with the first possible implementation of the first aspect, in the second possible implementation, before the current data block is written in the occupied space of the new data stream, and/or Before the fingerprint value of the current data block is written in the occupied space of the new data stream, the method further includes:

Finding a location of the data block of the n-1th layer included in the current data block in the new data stream. In combination with the first aspect, or in combination with the first and second possible implementations of the first aspect, in a third possible implementation, the previous one of the original data streams from the nth layer After the data between the end position of the data block and the end position of the sliding window is taken as the current data block, the method further includes:

Recording a sequence number of the current data block, a level corresponding to the current data block, a location of the current data block in the original data stream, and a location of the current data block in the new data stream Relationship, and saved in the location record table;

Finding a location of the data block of the n-1th layer included in the current data block in the new data stream, including:

According to the location of the current data block in the original data stream and the n-1th data block in the original a location in the data stream, in the location record table, locating a record of the n-1th data block included in the current data block in the original data stream;

Obtaining, in the record of the n-1th data block that is located, the position of the n-1th data block included in the current data block in the new data stream.

In conjunction with the first aspect, or in combination with the first to third possible implementations of the first aspect, in a fourth possible implementation, the method further includes:

In the process of parsing the new data stream into the original data stream, when the fingerprint data value is found in the new data stream, searching for the data block corresponding to the fingerprint value in the fingerprint dictionary in the block Index information in the dictionary;

Searching, according to the index information, a data block corresponding to the fingerprint value in the block dictionary, and replacing the fingerprint value with the found data in a space occupied by the new data stream Piece.

In a second aspect, the embodiment of the present invention further provides a data blocking device, including:

An acquisition module: configured to acquire a fingerprint value of a sliding window on the original data stream, where an initial starting position of the sliding window is the same as a starting position of the original data stream, and a length of the sliding window is a preset length; a module, configured to adopt a fingerprint value of the sliding window, and respectively modulo the first modulo value to the Nth modulo value, wherein the ith modulo value is extended to the left by a preset number of digits to obtain an i+th a modulus value, wherein i is greater than or equal to 1 and less than any natural number of the N, and the N is a total number of modulus values;

a blocking module, configured, for each modulus value, if the value of the modulo module after modulo is zero, starting from the end position of the previous data block of the nth layer to the sliding in the original data stream The data between the end positions of the windows is the current data block, and the current data block is the nth layer data block; wherein the n is greater than or equal to 1 and less than or equal to any natural number of the N:

a writing module, configured to store, in the fingerprint dictionary, a fingerprint value of a current data block generated by the blocking module, store the current data block into a blocking dictionary, and set a fingerprint value of the current data block And index information of the current data block in the block dictionary is stored in the fingerprint dictionary, and when the n is greater than 1, the current data block is written in a space occupied by the new data stream to cover All data blocks of the nth-1th layer included in the current data block in the new data stream, when the n is equal to 1, the current data block is sequentially written into a space occupied by the new data stream;

The writing module is further configured to: when the fingerprint value of the current data block is stored in the fingerprint dictionary, and when n is greater than 1, write the current data block in a space occupied by the new data stream Fingerprint a value to cover a fingerprint value of all n-1th data blocks included in the current data block in the new data stream, where the fingerprint value of the current data block is stored in the fingerprint dictionary and When n is equal to 1, the fingerprint value of the current data block is sequentially written into the space occupied by the new data stream;

a sliding module, configured to: after the writing module writes the current data block generated by the blocking module or the fingerprint value of the current data block into a space occupied by the new data stream, the sliding window is And sliding the preset length on the original data stream to obtain the fingerprint value of the sliding window on the original data stream, so that the modulo module, the writing module, and the sliding module repeatedly perform operations until the The starting position of the sliding window slides to the end of the original data stream.

With reference to the second aspect, in a first possible implementation manner, when the current data block includes a data block of an n-1th layer, an index information of the current data block in the block dictionary, according to the The index information of the data block of the n-1th layer included in the current data block is determined.

With reference to the second aspect, or in combination with the first possible implementation of the first aspect, in a second possible implementation, the apparatus further includes:

a positioning module, configured to: before the writing module writes the current data block in a occupied space of a new data stream, and/or write the current in the occupied space of the new data stream in the write module Before the fingerprint value of the data block, the location of the data block of the n-1th layer included in the current data block is found in the new data stream.

With reference to the second aspect, or in combination with the first and second possible implementations of the first aspect, in a third possible implementation, the apparatus further includes:

a recording module, configured to record, after the block module, data in the original data stream from an end position of a previous data block of the nth layer to an end position of the sliding window as a current data block a sequence number of the current data block, a level of the current data block, a location of the current data block in the original data stream, and a location of the current data block in a location of the new data stream And saved in the location record table;

The positioning module is configured to: according to a location of the current data block in the original data stream and a position of an n-1th layer data block in the original data stream, in the location record table, a location location Decoding the n-1th data block included in the current data block in the original data stream; acquiring the n-1th layer included in the current data block from the recorded record of the n-1th data block The location of the data block in the new data stream.

With reference to the second aspect, or in combination with the first to third possible implementations of the first aspect, in a fourth possible implementation, the apparatus further includes: a restoring module, in the process of parsing the new data stream into the original data stream, searching for a data block corresponding to the fingerprint value in the fingerprint dictionary when a fingerprint value is found in the new data stream Index information in the block dictionary;

The restoring module is further configured to: search, according to the index information, a data block corresponding to the fingerprint value in the block dictionary, where the fingerprint value is in a space occupied by the new data stream, The fingerprint value is replaced with the found data block.

The data blocking method and device provided by the embodiment of the present invention, the fingerprint values of the same sliding window on the original data stream are respectively modulo to a set of modulus values, that is, the first modulus value to the Nth modulus value, according to different modes. The modulo result of the value is divided into different levels of the original data stream. Therefore, for the fingerprint value of the same sliding window, the original data stream can be coarse-grained and fine-grained at the same time. For the generated data block, if the fingerprint value of the data block exists in the fingerprint dictionary, according to the level of the data block, it is determined whether the fingerprint value of the data block is sequentially written into the space occupied by the new data stream or is replaced by the data block. If the fingerprint value of the data block does not exist in the fingerprint dictionary, according to the level of the data block, it is determined whether the data block is sequentially written into the space occupied by the new data stream or the data is replaced. The next layer of data blocks included in the block, thereby avoiding the phenomenon that redundant data blocks and redundant data blocks are present in the new data stream compared with the original data stream, thereby realizing the generation of new data blocks with different granularities. The data stream, and the purpose of replacing the existing small data blocks with large data blocks in the new data stream, improves the deduplication rate of the new data stream. In this embodiment, each layer of the data block is a block directly generated on the original data stream, and each fingerprint value is a fingerprint value of the data block. It is not necessary to add an indication bit in the fingerprint value, indicating that the fingerprint value is a data block. The fingerprint value is also the fingerprint value of the fingerprint value, which reduces the overhead of the fingerprint value of the data block and improves the deduplication rate of the new data stream. In addition, since different modulo values generate different levels of data blocks, the generation of each layer of data blocks does not depend on the data blocks of other layers. Therefore, for the fingerprint values of the same sliding window, different levels of data blocks can be generated concurrently. DRAWINGS

1A is a flowchart of a data blocking method according to an embodiment of the present invention;

FIG. 1B is a schematic diagram of a sliding window according to an embodiment of the present invention; FIG.

2 is a schematic diagram of a three-level block diagram according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a block dictionary and a fingerprint dictionary according to an embodiment of the present invention; FIG. 4 is another schematic structural diagram of a block dictionary and a fingerprint dictionary according to an embodiment of the present invention; FIG. 5 is a schematic structural diagram of a data blocking device according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of another data blocking device according to an embodiment of the present invention. detailed description

The embodiment of the present invention can be applied to before the sending end sends the data stream to the receiving end, the sending end blocks the original data stream, forms the generated data block into a new data stream, and sends a new data stream to the receiving end, thereby saving transmission bandwidth. Improve transmission efficiency.

FIG. 1 is a flowchart of a data blocking method according to an embodiment of the present invention. In this embodiment, the original data stream is multi-layered by a set of modulus values: the first modulus value to the Nth modulus value. Where N is the total number of moduli values, N is the default value, and N is a natural number greater than 1.

As shown in FIG. 1A, the method provided in this embodiment includes:

Step 11: Slide the sliding window from the starting position of the original data stream to the end position of the original data stream, and the initial starting position of the sliding window is the same as the starting position of the original data stream. The length of the sliding window is a preset length.

Step 12: Obtain the fingerprint value of the sliding window on the original data stream, and use the fingerprint value of the sliding window to respectively modulate the first modulo value to the Nth modulo value. For each modulo value, if the value is Steps 13 are performed, and then step 14 is performed.

The relationship between the first modulo value and the Nth modulo value is as follows: The first modulo value pi is a preset value, and the first modulo value is extended to the left by a preset number of bits to obtain a second modulo value p2, After the two modulo values are extended to the left by the preset number of bits, the third modulo value p3 is obtained... The ith modulo value is extended to the left by the preset number of bits to obtain the i+1th modulo value, and so on. The modulus values pl, p2, ..., pN of different numbers of bits, where i is any natural number greater than or equal to 1 and less than N, and N is the total number of modulus values. For example, the modulus value pl=1100 is extended to the left by 2 bits to obtain the modulus value p2=01 1100, and the modulus value p2 is further extended to the left by 1 bit to obtain the modulus value p3=1011100. In the same way, p4, , pN can be obtained.

Step 14: Slide the sliding window on the original data stream to the end position by a preset length, obtain the fingerprint value of the sliding window on the original data stream, and repeat steps 12 to 14 until the starting position of the sliding window slides to the original position. The end of the data stream.

As shown in FIG. 1B, the sliding window is slid from the starting position of the original data stream toward the end position of the original data stream by a preset length. The fingerprint value of the current sliding window is used to modulate the first modulo value to the Nth modulo value respectively, and after performing step 13, the sliding window is slid to a preset length in the direction of the end position of the original data stream. Continue to modulo the first modulo value to the Nth modulo value by using the fingerprint value of the next sliding window, and perform step 13 until the starting position of the sliding window slides to the end position of the original data stream. The fingerprint value of the data of the original data stream in the sliding window is called the fingerprint value of the sliding window. The fingerprint value of the data may be an identifier of the data, which is used to identify the data, and the fingerprint values of different data are different.

Using the fingerprint value of the sliding window, the first modulo value to the Nth modulo value are respectively modulo, and when step 13 is performed, the order from the first modulo value to the Nth modulo value may be sequentially performed. Each modulo value is modulo, and step 13 is performed when the value after modulo is zero; or modulo N values may be simultaneously modulo, and then from the first modulo value to the Nth modulo value In the order of the steps, step 13 is performed when the value after modulo is zero.

If the fingerprint value of the sliding window is zero after modulo the nth modulus, perform step 13. A data block is generated in step 13 and it is determined whether the data block or the fingerprint value of the data block is written into the new data stream footprint. Step 13 includes the following operations.

Step 13 includes: as the current data block, the data in the original data stream from the end position of the previous data block of the nth layer to the end position of the sliding window is the current data block, where the current data block is the nth data block; n is any natural number greater than or equal to 1 and less than or equal to N.

In other words, in the above step 12, if the fingerprint value of the sliding window is zero after the modulo of the nth modulus, a data block is generated from the original data stream, and the starting position of the current data block is the original data. The end position of the previous data block of the nth layer in the stream, and the end position of the current data block is the end position of the sliding window. The data block generated for the nth modulus value is called the data block of the nth layer. The number of bits of the modulus extension can be customized, and the number of bits of the current modulus extension affects the length of the next block of data.

The step 13 further includes: when the fingerprint value of the current data block is not stored in the fingerprint dictionary, storing the current data block into the block dictionary, and the fingerprint value of the current data block and the index information of the current data block in the block dictionary Stored in the fingerprint dictionary; if n is greater than 1, write the current data block in the space occupied by the new data stream to cover all the n-1th data blocks included in the current data block in the new data stream, otherwise the current data block The data blocks are sequentially written to the space occupied by the new data stream.

The block dictionary is used to store data blocks generated on the original data stream, and the fingerprint dictionary is used to store the fingerprint value of the data block stored in the block dictionary and the index information of the data block in the block dictionary. For example, the index information of the data block in the block dictionary may include the length of the data block and the location information of the data block in the block dictionary.

After generating a data block from the original data stream, it is determined whether the fingerprint value of the current data block is stored in the fingerprint dictionary. If the fingerprint dictionary does not store the fingerprint value of the current data block, the new data stream and The current data block is not stored in the block dictionary, and the current data block needs to be written in the new data stream and the block dictionary; if the fingerprint value of the current data block is stored in the fingerprint dictionary, it can be determined that the new data stream and the block dictionary have been The current data block is stored, and the current data block does not need to be written in the new data stream and the block dictionary.

The nth modulus value is obtained by extending the fixed number of bits from the n-1th modulus value to the left. For the fingerprint value of the same sliding window, if the value of the fingerprint value of the sliding window is zero after the modulo of the nth modulus value, the value of the fingerprint value of the sliding window is modulo the value of the n-1th modulus value. Also zero. The fingerprint value of the sliding window is zero after modulo the n-1th modulus, and the value of the fingerprint value of the sliding window is not necessarily zero after the modulo of the nth modulus. For the fingerprint value of the same sliding window, after the n-1th data block is generated, the nth layer data block is not necessarily generated, but when the nth layer data block is generated, the n-1th layer data block is generated. Must have been produced. Therefore, when the sliding window is moved from the original data stream starting position to the ending position, the starting position of the nth layer data block is closer to the original data stream than the starting position of the n-1th data block. The starting position is not farther than the starting position of the data block of the n-1th layer. The length of the data block of the nth layer is greater than or equal to the length of the data block of the n-1th layer, and the data block of the nth layer includes one or more data blocks of the n-1th layer.

In the case that the fingerprint value of the current data block is not stored in the fingerprint dictionary, if the current data block is at a level n greater than 1, the current data block includes one or more n-1th data blocks, in order to avoid new data streams. The fingerprint value of the redundant data block and the redundant data block appear in the space, and the current data block is written into the space occupied by the new data stream, so as to cover all the n-1th data blocks included in the current data block in the new data stream. . Wherein, before the current data block is written, the location of the current data block in the new data stream stores all the n-1th data blocks included in the current data block; after the current data block is written, the current data block is overwritten. All the n-1th data blocks included in the current data block in the new data stream, therefore, the current data block is stored in the location of the current data block in the new data stream.

If the current data block is at the level n equal to 1, since the current data block is the data block of the first layer, the data block of other layers is not included in the range, so the current data block is directly written to the end of the new data stream.

When the level n of the current data block is greater than 1, the data block of the n-1th layer included in the current data block may be found in the new data before the current data block is written in the occupied space of the new data stream. The position in the stream to determine where to write the current data block in the footprint of the new data stream. The data block of the n-1th layer included in the current data block of the nth layer can be located in the new data stream by the following method.

First, after generating a data block, the sequence number of the data block, the level of the data block, and the correspondence between the locations of the original data streams are recorded. The above correspondence can be recorded in the location record table. For example, after the third data block is generated, because the third data block is the data block of the second layer, it is the third one. The correspondence of the data block records is (No. 3, Level 2, the position in the original data stream). The sequence number of the data block indicates the sequence in which the data blocks are generated. The smaller the sequence number, the earlier the time is generated. After the fingerprint value of the data block/the data block is written in the new data stream, the position of the data block/the fingerprint value of the data block in the new data stream is increased in the above correspondence. For example, after the fingerprint value of the third data block/third data block is written in the new data stream, the third data block/third data block is added to the correspondence relationship recorded for the third data block. The location of the fingerprint value in the new data stream. At this time, the correspondence recorded for the third data block is (No. 3, Level 2, position in the original data stream, position in the new data stream). Wherein, the location of the fingerprint value of the data block/data block in the data stream may be represented by the starting position of the data block/data block in the data stream and the length of the data block, or may be data block/data The fingerprint value of the block is represented by the start and end positions in the data stream. By the position of the recorded fingerprint/data block fingerprint value in the data stream, the data block/the data block's fingerprint value can be found in the data stream.

When the fingerprint value of the current data block is not stored in the fingerprint dictionary and the level n of the current data block is greater than 1, according to the position of the current data block in the original data stream and the position of the n-1th data block in the original data stream, A record of the n-1th data block included in the current data block in the original data stream is located in the location record table. Obtaining, in the record of the n-1th data block that is located, the location of the n-th layer data block included in the current data block in the new data stream.

The data block of the n-1th layer included in the current data block is referred to as the data block to be replaced of the n-1th layer. After a data block is generated in the original data stream, it is possible to write the data block in the new data stream, or it may write the fingerprint value of the data block. Therefore, in the location record table, the location of the data block to be replaced on the n-1th layer in the location of the new data stream may be the location of the data block to be replaced in the n-1th layer in the new data stream, or Is the location of the fingerprint value of the data block to be replaced on the n-1th layer in the new data stream. From the record of the to-be-replaced data block of the n-1th layer that is located, the fingerprint value of the to-be-replaced data block/n-th layer of the n-1th layer to be replaced may be obtained in the new data stream. s position. After obtaining the location of the fingerprint value of the to-be-replaced data block/n-1th layer of the to-be-replaced data block of the n-1th layer in the new data stream, according to the acquired location, the nth layer is used in the new data stream. The current data block replaces the fingerprint value of the to-be-replaced data block/n-1th layer of the to-be-replaced data block of the n-1th layer, and the record of the n-1th layer of the to-be-replaced data block is deleted in the location record table.

The step 13 further includes: when the fingerprint value of the current data block is stored in the fingerprint dictionary, if n is greater than 1, the fingerprint value of the current data block is written in a space occupied by the new data stream to cover the new data. In the stream The fingerprint value of all n-1th data blocks included in the current data block, otherwise the fingerprint value of the current data block is sequentially written into the space occupied by the new data stream.

If the fingerprint value of the current data block exists in the fingerprint dictionary, it can be determined that the current data block is already stored in the block dictionary and the new data stream. If the fingerprint value of the current data block exists in the fingerprint dictionary, and the current data block is a layer 1 data block, the fingerprint value of the current data block is written directly at the end of the last data block of the new data stream.

In the case where the level n of the current data block is greater than 1, the current data block includes one or more n-1th data blocks. Therefore, when the current data block is generated, if the fingerprint value of the current data block is already stored in the fingerprint dictionary, the fingerprint values of all the n-1th data blocks in the current data block range are already written in the new data stream. In order to avoid the fingerprint value of redundant data blocks and redundant data blocks in the new data stream, the fingerprint value of the current data block is written in the space occupied by the new data stream to cover all the current data blocks included in the new data stream. The fingerprint value of the data block of the n-1th layer, thereby saving the transmission bandwidth of the new data stream. Wherein, before writing the fingerprint value of the current data block, the location of the current data block in the new data stream stores the fingerprint value of all the n-1th data blocks included in the current data block; After the fingerprint value, the fingerprint value of the current data block covers the fingerprint value of all the n-1th data blocks included in the current data block in the new data stream, and the current data block in the new data stream stores the current data. The fingerprint value of the block.

When the level n of the current data block is greater than 1, the data block of the n-1th layer included in the current data block may be found in the new data block before the fingerprint value of the current data block is written in the occupied space of the new data stream. The location in the data stream to determine the location of the fingerprint value written to the current data block in the footprint of the new data stream. The fingerprint value of the n-1th data block within the current data block range of the nth layer can be located in the new data stream by the following method. According to the position of the current data block in the original data stream and the position of the n-1th data block in the original data stream, in the location record table, the n-1th data block included in the current data block in the original data stream is located. record of. Obtaining, in the record of the n-1th data block that is located, the location of the n-1th data block included in the current data block in the new data stream. The data block of the n-1th layer included in the current data block is referred to as the data block to be replaced of the n-1th layer. In the recorded record of the n-1th layer to be replaced, the location of the fingerprint value of the n-1th layer of the to-be-replaced data block in the new data stream is obtained. After obtaining the location of the fingerprint value of the data block to be replaced on the n-1th layer in the new data stream, replacing the n-1th layer with the fingerprint value of the current data block in the new data stream according to the acquired location The fingerprint value of the data block is replaced, and the record of the n-1th layer of the data block to be replaced is deleted in the location record table. When the start position of the sliding window slides to the end position of the original data stream, the segmentation of the original data stream is ended, and the new data stream at this time is the substitute data stream of the original data stream. The receiving end can combine the block dictionary and the fingerprint dictionary to parse the new data stream sent by the sending end to obtain the original data stream. In the process of parsing the new data stream into the original data stream, if the fingerprint value is found in the new data stream, the fingerprint dictionary searches for the index information of the data block corresponding to the fingerprint value in the block dictionary. According to the index information, the data block corresponding to the fingerprint value is searched in the block dictionary, and the fingerprint value is replaced with the searched data block in the space occupied by the new data stream.

In the data blocking method provided in this embodiment, the fingerprint values of the same sliding window on the original data stream are respectively moduloed to a set of moduli values, that is, the first modulo value to the Nth modulo value, according to different modulo values. The modulo result performs different levels of partitioning on the original data stream. Therefore, for the fingerprint value of the same sliding window, the original data stream can be coarse-grained and fine-grained at the same time. For the generated data block, if the fingerprint value of the data block exists in the fingerprint dictionary, according to the level of the data block, it is determined whether the fingerprint value of the data block is sequentially written into the space occupied by the new data stream or the data block is replaced. The fingerprint value of the data block of the next layer is included; if the fingerprint value of the data block does not exist in the fingerprint dictionary, according to the level of the data block, it is determined whether the data block is sequentially written into the space occupied by the new data stream or replaced The next block of data blocks included in the data block, thereby avoiding the phenomenon that the fingerprint values of the redundant data blocks and the redundant data blocks appear in the new data stream compared with the original data stream, and the data blocks with different granularities are realized. The purpose of generating a new data stream and replacing the existing small data block with a large data block in the new data stream improves the deduplication rate of the new data stream. In this embodiment, each layer of the data block is a block directly generated on the original data stream, and each fingerprint value is a fingerprint value of the data block. It is not necessary to add an indication bit in the fingerprint value, indicating that the fingerprint value is a data block. The fingerprint value is also the fingerprint value of the fingerprint value, which reduces the overhead of the fingerprint value of the data block and improves the deduplication rate of the new data stream. In addition, since different modulo values generate different levels of data blocks, the generation of each layer of data blocks does not depend on the data blocks of other layers. Therefore, for the fingerprint values of the same sliding window, different levels of data blocks can be generated concurrently.

Further, in order to save the storage space of the block dictionary, when the fingerprint value of the current data block is not stored in the block dictionary, it is determined whether the current data block includes the n-1th data block to determine whether it is in the block dictionary. Write the current data block. If the current data block includes the data block of the n-1th layer, storing the fingerprint value of the current data block and the index information of the current data block in the block dictionary into the fingerprint dictionary, where The index information of the current data block in the block dictionary is determined according to index information of a data block of the n-1th layer included in the current data block. If the current data block does not include the first a data block of the n-1 layer, storing the current data block into the block dictionary, storing the fingerprint value of the current data block and the index information of the current data block in the block dictionary to the In the fingerprint dictionary.

A three-level block diagram as shown in Figure 2. The blocks in Figure 2 use modulo values p l, p2 and p3. Pl=10, expand pi to the left by 1 bit, and get p2=110. Extend pi to the left by two bits to get p3=01110.

The sliding window slides from left to right on the original data stream, and the fingerprint value in the sliding window is 111010. The fingerprint value of the sliding window is 0 after modulo P i , and a block of the first layer is generated, which is recorded as block 1. At this time, the fingerprint value of the sliding window is not 0 after the modulo values of p2 and p3. The fingerprint of the block 1 is not stored in the fingerprint dictionary. As shown in FIG. 3, the block 1 is stored in the block dictionary, and the fingerprint value of the block 1 and the block 1 are stored in the block dictionary in the fingerprint dictionary. Index information. The index information of the block in the block dictionary includes the address, offset, and length in the block dictionary. Then, block 1 is written to the new data stream. For example, the address of block 1 in the block dictionary is the location of the entry in the block dictionary that stores the block 1 data.

The sliding window continues to slide to the right on the original data stream, and the sliding window has a fingerprint value of 010110. The fingerprint value of the sliding window has a value of 0 after modulo, and a block of the first layer is generated, which is recorded as block 2. The value of the fingerprint value of the sliding window is also 0 after modulo p2, and a block of the second layer is generated, which is recorded as block 3. The fingerprint value of the sliding window is not 0 after modulo p3. The fingerprint of the block 2 is not stored in the fingerprint dictionary. As shown in FIG. 3, the block 2 is stored in the block dictionary, and the fingerprint value of the block 2 and the block 2 are stored in the block dictionary in the fingerprint dictionary. 2 Start address, offset and length, then, block 2 is the last block in the new data stream, that is, the end of block 1. After generating the block 3, the fingerprint of the block 3 is not stored in the fingerprint dictionary, the block 3 is stored in the block dictionary, and the fingerprint value of the block 3 and the block 3 are stored in the block dictionary in the fingerprint dictionary. Address, offset, and length, then, block 1 and block 2 of layer 1 are replaced with block 3 of layer 2 in the new data stream.

The sliding window continues to slide to the right on the original data stream. The fingerprint value of the sliding window is 001110. The fingerprint value of the sliding window is 0 for Pl, p2 and p3, and the block 4 of the first layer, the block 4 of the second layer, and the block 5 of the third layer are successively generated. The partition 4 of the first layer and the partition 4 of the second layer are the same partition, and therefore, only the partition of the first layer is stored, which is denoted as the partition 4. After the block 4 is generated, the fingerprint value of the block 4 is not yet stored in the fingerprint dictionary. As shown in FIG. 3, the block 4 is stored in the block dictionary, and the fingerprint value and the score of the block 4 are saved in the fingerprint dictionary. The starting address, offset and length of block 4 in the blocking dictionary, then block 4 is written to the end of the last block in the new data stream, block 1. After the block 5 is generated, the fingerprint value of the block 5 is not yet stored in the fingerprint dictionary, and the block 5 is stored in the block dictionary.

The fingerprint value of the block 5 and the start address and offset of the block 5 in the block dictionary are saved in the fingerprint dictionary. The quantity and length, then, the block 3 of the second layer and the block 4 are replaced with the block 5 of the third layer in the new data stream. The sliding window continues to slide to the right on the original data stream. If the block 3 is generated, it is determined that the fingerprint value of the block 3 has been stored in the fingerprint dictionary and the block 3 is the block of the second layer, which can be determined in the new data stream. The data block of the first layer included in the range occupied by the current block 3 has been replaced with the fingerprint value, and the block 1 of the first layer in the range of the block 3 in the original data stream is replaced by the fingerprint value of the block 3. Block 2. If the result is block 5, it is determined that the fingerprint value of the block 5 has been stored in the fingerprint dictionary and the block 3 is the block of the second layer, and it can be determined that in the new data stream, the range occupied by the current block 5 is included. The data block of the second layer has been replaced with the fingerprint value, and the block 3 and the block 4 of the second layer in the range of the block 5 in the original data stream are replaced by the fingerprint value of the block 5. If the block 1 or 2 of the first layer is generated, since the fingerprint value of the block 1 or the block 2 is already stored in the fingerprint dictionary, and the block 1 or the block 2 is the block of the first layer, directly The fingerprint value of Block 1 or Block 2 is written at the end of the last block of the new data stream.

Further, in order to save the storage space of the block dictionary, if the block 3 is generated, as can be seen from FIG. 2, the block 3 of the second layer is composed of the block 1 and the block 2 of the first layer, in the block dictionary. Block 3 has been stored and block 3 can be stored repeatedly in the block dictionary. As shown in Fig. 4, the offset of the block 1 is 0, the block 2 is stored after the block 1, and the offset of the block 2 is the length of the block 1. For the block 3 of the second layer, the address of the block 3 is the same as the address of the block 1, and is also the same as the address of the block 2, and the offset of the block 3 is the offset of the block 1. The length of the partition 3 is the sum of the length of the partition 1 and the length of the partition 2. Similarly, the block 5 of the third layer is composed of the block 3 and the block 4 of the second layer, and the block 5 is already stored in the block dictionary, and the block 5 can be repeatedly stored in the block dictionary. The offset of the partition 5 is the offset of the partition 3. The address of the partition 5 is the same as the address of the partition 3, and the length of the partition 5 is the sum of the length of the partition 3 and the length of the partition 4.

FIG. 5 is a schematic structural diagram of a data blocking device according to an embodiment of the present invention. As shown in FIG. 5, the apparatus provided in this embodiment includes: an acquisition module 56, a modulo module 51, a blocking module 52 and a writing module 53, and a sliding module 50.

The obtaining module 56 is configured to obtain a fingerprint value of the sliding window on the original data stream, where the initial starting position of the sliding window is the same as the starting position of the original data stream, and the length of the sliding window is a preset length.

The modulo module 51 is configured to perform modulo adjustment on the first modulo value to the Nth modulo value by using the fingerprint value of the sliding window, where the ith modulo value is extended to the left by a preset number of bits. The i+1th modulus value, wherein i is greater than or equal to 1 and less than any natural number of the N, and the N is a total number of modulus values.

a blocking module 52, configured, for each modulus value, if the value of the modulo module after modulo is zero, starting from the end position of the previous data block of the nth layer in the original data stream to the Sliding window knot The data between the bundle positions is the current data block, and the current data block is the n-th data block; wherein the n is greater than or equal to 1 and less than or equal to any natural number of the N:

The writing module 53 is configured to: when the fingerprint value of the current data block generated by the blocking module is not stored in the fingerprint dictionary, store the current data block into the blocking dictionary, and use the fingerprint value of the current data block to The index information of the current data block in the block dictionary is stored in the fingerprint dictionary; if the n is greater than 1, the current data block is written in a space occupied by the new data stream to cover the current data block All the data blocks of the n-1th layer included in the current data block in the new data stream, otherwise the current data block is sequentially written into the space occupied by the new data stream;

The writing module 53 is further configured to: when the fingerprint value of the current data block is stored in the fingerprint dictionary, if n is greater than 1, write the current data block in a space occupied by the new data stream a fingerprint value, to cover a fingerprint value of all n-1th data blocks included in the current data block in the new data stream, or sequentially writing the fingerprint value of the current data block to the new data stream Space;

a sliding module 50, configured to: after the writing module writes the current data block generated by the blocking module or the fingerprint value of the current data block into a space occupied by the new data stream, the sliding window And sliding the preset length on the original data stream to the end position, and acquiring the fingerprint value of the sliding window on the original data stream, so that the modulo module 51, the writing module 53 and the sliding module 50 repeatedly perform operations. Until the starting position of the sliding window slides to the end position of the original data stream.

Optionally, the writing module 53 is further configured to: if the current data block includes a data block of an n-1th layer, the fingerprint value of the current data block and the current data block in the block dictionary The index information is stored in the fingerprint dictionary, wherein the index information of the current data block in the block dictionary is determined according to index information of the n-1th data block included in the current data block. .

Optionally, the writing module 53 is further configured to: if the current data block does not include the data block of the n-1th layer, store the current data block into the block dictionary, and use the current data The fingerprint value of the block and the index information of the current data block in the block dictionary are stored in the fingerprint dictionary.

In the data blocking device provided by the embodiment, the fingerprint module of the same sliding window in the original data stream respectively modulates the first modulus value to the Nth modulus value of a set of modulus values, and the blocking module is different according to different parameters. The modulo modulo result performs different levels of sharding on the original data stream. Therefore, for the fingerprint value of the same sliding window, the original data stream can be coarse-grained and fine-grained at the same time. For the generated data block, if the fingerprint value of the data block exists in the fingerprint dictionary, the writing module determines, according to the level of the data block, whether the fingerprint value of the data block is sequentially written into the space occupied by the new data stream or replaces the data. Block includes The fingerprint value of the next layer of data block; if the fingerprint value of the data block does not exist in the fingerprint dictionary, the write module determines, according to the level of the data block, whether to sequentially write the data block to the space occupied by the new data stream or replace The data block includes the next layer of data blocks, thereby avoiding the phenomenon that the fingerprint values of the redundant data blocks and the redundant data blocks appear in the new data stream compared with the original data stream, and the data blocks with different granularities are realized. The purpose of generating a new data stream and replacing the existing small data block with a large data block in the new data stream improves the deduplication rate of the new data stream. In this embodiment, each layer of the data block is a block directly generated on the original data stream, and each fingerprint value is a fingerprint value of the data block. It is not necessary to add an indication bit in the fingerprint value, indicating that the fingerprint value is a data block. The fingerprint value is also the fingerprint value of the fingerprint value, which reduces the overhead of the fingerprint value of the data block and improves the deduplication rate of the new data stream. In addition, since different moduli values generate different levels of data blocks, the generation of each layer of data blocks does not depend on data blocks of other layers. Because of this, for the fingerprint value of the same sliding window, different levels of data can be generated concurrently. Piece.

As shown in FIG. 6, the apparatus of FIG. 5 may further include: a positioning module 54 and a recording module 55.

a recording module 55, configured to, after the blocking module 52, data in the original data stream from an end position of a previous data block of the nth layer to an end position of the sliding window as a current data block Recording a sequence number of the current data block, a level between the current data block, a location of the current data block in the original data stream, and a location of the current data block between the new data stream Correspondence relationship, and saved in the location record table;

a positioning module 54, configured to: before the writing module 53 writes the current data block in a occupied space of a new data stream, and/or the writing module 53 writes in the occupied space of the new data stream Before the fingerprint value of the current data block is described, the location of the data block of the n-1th layer included in the current data block is found in the new data stream.

Further, the positioning module 54 is specifically configured to record at the location according to a location of the current data block in the original data stream and a location of an n-1th data block in the original data stream. In the table, the record of the n-1th data block included in the current data block in the original data stream is located; and the current data block is included in the record of the n-1th data block that is located The location of the n-1th data block in the new data stream.

In order to restore the original data stream based on the new data stream, the apparatus shown in Figures 5 and 6 may further include: a restore module.

a restoring module, in the process of parsing the new data stream into the original data stream, if a fingerprint value is found in the new data stream, searching for a data block corresponding to the fingerprint value in the fingerprint dictionary Place Defining index information in the block dictionary;

The various modules shown in Figures 5 and 6 can be implemented using a processor. Optionally, the apparatus shown in FIG. 5 and FIG. 6 may further include a storage module, configured to store the block dictionary and the fingerprint dictionary. Optionally, the storage module can be implemented by using a memory.

A person skilled in the art can understand that all or part of the steps of implementing the above method embodiments may be implemented by hardware related to program instructions, and the foregoing program may be stored in a computer readable storage medium, when executed, The steps of the foregoing method embodiments are performed; and the foregoing storage medium includes: various media that can store program codes, such as ROM, RAM, disk or optical disk.

It should be noted that the above embodiments are only for explaining the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that: The technical solutions described in the foregoing embodiments are modified, or some of the technical features are equivalently replaced; and the modifications or substitutions do not deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

A data blocking method, comprising:

When the fingerprint value of the current data block is stored in the fingerprint dictionary and when n is greater than 1, the fingerprint value of the current data block is written in a space occupied by the new data stream to cover the new a fingerprint value of all n-1th data blocks included in the current data block in the data stream, when the fingerprint value of the current data block is stored in the fingerprint dictionary, and when n is equal to 1, the The fingerprint value of the current data block is sequentially written into the space occupied by the new data stream;

2. The method according to claim 1, wherein the current data block is stored in a block dictionary, and the fingerprint value of the current data block and the current data block are in the block dictionary The index information is stored in the fingerprint dictionary, including:

When the current data block includes a data block of the n-1th layer, the fingerprint value of the current data block is The index information of the current data block in the block dictionary is stored in the fingerprint dictionary, wherein the index information of the current data block in the block dictionary is included according to the current data block. The index information of the n-1 layer data block is determined;

The method according to claim 1 or 2, characterized in that before the current data block is written in the occupied space of the new data stream, and/or written in the occupied space of the new data stream Before describing the fingerprint value of the current data block, it also includes:

Finding a location of the data block of the n-1th layer included in the current data block in the new data stream.

4. The method of claim 3, wherein:

After the data between the end position of the previous data block of the nth layer and the end position of the sliding window in the original data stream is used as the current data block, the method further includes:

And locating the location in the original data stream in the location record table according to a location of the current data block in the original data stream and a location of an n-1th data block in the original data stream a record of the n-1th data block included in the current data block;

5. The method according to claim 1 or 2, further comprising:

Searching, according to the index information, a data block corresponding to the fingerprint value in the block dictionary, and replacing the fingerprint value with the found data in a space occupied by the new data stream Piece. 6. A data blocking device, comprising:

The writing module is further configured to: when the fingerprint value of the current data block is stored in the fingerprint dictionary, and when n is greater than 1, write the current data block in a space occupied by the new data stream a fingerprint value to cover a fingerprint value of all n-1th data blocks included in the current data block in the new data stream, when the fingerprint value of the current data block is stored in the fingerprint dictionary and When the n is equal to 1, the fingerprint value of the current data block is sequentially written into the space occupied by the new data stream;

The device according to claim 6, wherein: when the current data block includes a data block of an n-1th layer, index information of the current data block in the block dictionary, according to the The index information of the data block of the n-1th layer included in the current data block is determined.

The device according to claim 6 or 7, further comprising: a positioning module, configured to: before the writing module writes the current data block in a occupied space of a new data stream, and/or write the current in the occupied space of the new data stream in the write module Before the fingerprint value of the data block, the location of the data block of the n-1th layer included in the current data block is found in the new data stream.

9. The device according to claim 8, further comprising:

The positioning module is specifically configured to: according to a location of the current data block in the original data stream and a location of the n-1th data block in the original data stream, in the location record table, a record of the n-1th data block included in the current data block in the original data stream; and obtaining an n-1th included in the current data block from the recorded record of the n-1th data block The location of the layer data block in the new data stream.

The device according to claim 6 or 7, further comprising:

a restoring module, in the process of parsing the new data stream into the original data stream, searching for a data block corresponding to the fingerprint value in the fingerprint dictionary when a fingerprint value is found in the new data stream Index information in the block dictionary;