CN102843403A - File processing method based on distributed file system, system, and client - Google Patents
File processing method based on distributed file system, system, and client Download PDFInfo
- Publication number
- CN102843403A CN102843403A CN 201110172216 CN201110172216A CN102843403A CN 102843403 A CN102843403 A CN 102843403A CN 201110172216 CN201110172216 CN 201110172216 CN 201110172216 A CN201110172216 A CN 201110172216A CN 102843403 A CN102843403 A CN 102843403A
- Authority
- CN
- China
- Prior art keywords
- node
- data block
- file
- memory device
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a file processing method based on a distributed file system, the system, and a client. The method comprises the following steps: the client sends a file writing request to a meta node; the client receives a file identifier, a data block identifier of each data block, the copy quantity of data blocks and a selectivity factor from the meta node; the client performs hashing operation on the file identifier of the file, the data block identifiers of the data blocks and the selectivity factor to obtain a hashing value; the client finds a consistent hashing ring according to the hashing value, obtains the value of a minimal virtual node greater than the hashing value and clockwise selects virtual nodes consistent with the copies in quantity from the virtual node on the consistent hashing ring; and the client sends the data blocks to different storage devices for storage when the selected virtual nodes meet the pre-condition. According to the invention, the occupation of the storage space of the meta node is reduced, the location speed during data writing can be improved, and the load balance of the system is implemented.
Description
Technical field
The application relates to computer memory technical field, particularly relates to a kind of document handling method based on distributed file system, system and client.
Background technology
Distributed file system (Distributed File System) is meant that the physical store resource of file system management not necessarily directly is connected on the local node, but links to each other with node through computer network.The design of distributed file system is based on Client.Referring to Figure 1A,, mainly comprise client, metadata node and data memory node for a kind of typical distributed file system configuration diagram.Wherein, client (Client) is deployed on the service application service device usually, and business application is sent data access request through the place client to the distributed file system cluster; Data memory node (chunk node) is used for the store files data, wherein possibly be made up of some memory devices on each data memory node; Facility information table and file layout allocation table have been preserved on the metadata node in the distributed file system cluster (meta node); Wherein be used to preserve the attribute information of the memory device of forming each data memory node in the facility information table; For example, memory capacity, active volume etc. have been preserved the memory device at each data block place of forming each file in the file layout allocation table; Shown in Figure 1B; Its file 1 is made up of N data block, is example with data block 1, needs the data block id (sign) of this data block of record, the back end id and the memory device id of data block place back end.
Prior art; When carrying out the file write operation; Metadata node need be searched the active volume of each memory device in the facility information table, selecting suitable storage equipment, after the memory device that client is selected according to metadata node is accomplished file and write; With the file identification of this document, and each data block records in the file layout allocation table with the corresponding relation that belongs to memory device; When carrying out the file read extract operation; Metadata node is according to file identification locating file layout allocation table; The positional information at each data block place of the composition this document that gets access to is returned to client, to the memory device of correspondence, read file data according to this positional information by client.
The inventor finds in the research process to prior art; When prior art will realize based on the file read-write operation of distributed file system; Need on metadata node, pass through the stored position information of the data block of each file of file layout assignment table record, when the distributed file system scale enlarges, quantity of documents will increase fast; The corresponding also corresponding increase of file layout allocation table, therefore having increased metadata node is the shared disk space of storage file layout allocation table; And when the file layout allocation table increased, the time that this document layout allocation table is carried out I/O (I/O) operation had been reduced the readwrite performance of file thus also with elongated.
Summary of the invention
The application embodiment provides a kind of document handling method based on distributed file system, system and client; To solve the existing read-write operation that carries out distributed document according to the file layout allocation table; Cause taking disk storage space, the problem that the file read-write performance is not high.
In order to solve the problems of the technologies described above, the application embodiment discloses following technical scheme:
A kind of document handling method based on distributed file system; Said system comprises metadata node and the client and the data memory node of communicating by letter respectively with said metadata node; Each data memory node is made up of at least one memory device; Preserve the consistency Hash ring based on dummy node in the client, wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity, and each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node; According to the value that each dummy node distributes dummy node that adds up to of said dummy node, said method comprises:
Client is sent file to metadata node and is write request;
Receive metadata node and be the file identification of said file allocation according to what said file write that request returns; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Form each data block of said file for order, the data block sign of the file identification of said file, said data block and selectivity factor are carried out Hash operation obtain cryptographic hash;
Search said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
When selected dummy node belongs to different storage devices respectively, and the user mode of said different storage devices is when being upstate, data block sent on the said different storage devices store.
A kind of distributed file system comprises: metadata node and the client and the data memory node of communicating by letter respectively with said metadata node, and each data memory node is made up of at least one memory device,
Said client; Be used to preserve consistency Hash ring based on dummy node; Wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity; Each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node, according to the value that each dummy node distributes dummy node that adds up to of said dummy node; And send file to metadata node and write request;
Said metadata node; Be used for writing request and be the file identification of said file allocation to what said client was returned according to said file; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Said client also is used for forming each data block of said file for order, the data block sign of the file identification of said file, said data block and selectivity factor is carried out Hash operation obtain cryptographic hash; Search said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash; When selected dummy node belongs to different storage devices respectively, and the user mode of said different storage devices sends to data block on the said different storage devices when being upstate;
Said memory device is used for the data block that said client is sent is preserved.
A kind of client is applied in the distributed file system, and said system comprises metadata node, and said metadata node is communicated by letter with data memory node with said client respectively, and each back end is made up of at least one memory device, and said client comprises:
Preserve the unit; Be used to preserve consistency Hash ring based on dummy node; Wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity; Each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node, according to the value that each dummy node distributes dummy node that adds up to of said dummy node;
Transmitting element is used for sending file to metadata node and writes request;
Receiving element; Be used to receive metadata node and be the file identification of said file allocation according to what said file write that request returns; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Arithmetic element is used for forming each data block of said file for order, the data block sign of the file identification of said file, said data block and selectivity factor is carried out Hash operation obtain cryptographic hash;
Acquiring unit; Be used for searching said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
Transmitting element also is used for belonging to different storage devices respectively when selected dummy node, and the user mode of said different storage devices is when being upstate, data block is sent on the said different storage devices store.
Can find out by the foregoing description; Send file by client to metadata node among the application embodiment and write request; Receive metadata node and write the file identification that request is returned for this document distribution according to file; And copy amount and the initialization value of forming data block sign, the data block of each data block of said file be zero selectivity factor, forms each data block of said file for order, the data block sign of the file identification of file, data block and selectivity factor carried out Hash operation obtain cryptographic hash; Search the consistency Hash ring of preserving on the client according to cryptographic hash; Obtain value, begin on consistency Hash ring, to select the dummy node consistent clockwise from this dummy node, when selected dummy node belongs to different storage devices respectively with copy amount greater than the minimum dummy node of this cryptographic hash; And when the user mode of different storage devices is upstate, data block is sent on the different storage devices store.In the distributed file system of the application embodiment; Owing to preserved consistency Hash ring in the client; Therefore when execution writes file operation; Need not metadata node and preserve the distribution situation of file on the memory device of each memory node through the file layout allocation table in advance; File allocation and the positioning action originally on metadata node, carried out are moved to client executing, and metadata node only needs the selectivity factor of transaction file data block, has therefore reduced taking the metadata node memory space; Because each client only need be when self carries out write operation be carried out the location of memory device according to consistency Hash ring; Rather than position by the request that writes of metadata node to all clients in the system, therefore can improve the locating speed when writing data; And, thereby through the user mode of the corresponding memory device of each dummy node on the consistency Hash ring being judged self adaptation carries out the selection of memory device, realized thus making system reach load balancing according to the residual capacity of disk.
Description of drawings
In order to be illustrated more clearly in the application embodiment or technical scheme of the prior art; To do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below; Obviously; For those of ordinary skills, under the prerequisite of not paying creative work property, can also obtain other accompanying drawing according to these accompanying drawings.
Figure 1A is a kind of typical distributed file system configuration diagram;
Figure 1B is the structural representation of file layout allocation table of the prior art;
Fig. 2 is the first embodiment flow chart of the application based on the document handling method of distributed file system;
Fig. 3 A is the second embodiment flow chart of the application based on the document handling method of distributed file system;
Fig. 3 B is the chain hoist pennants of the memory device among the application embodiment;
Fig. 3 C is the structural representation of consistency Hash ring among the application embodiment;
Fig. 4 is the three embodiment flow chart of the application based on the document handling method of distributed file system;
Fig. 5 is the four embodiment flow chart of the application based on the document handling method of distributed file system;
Fig. 6 is the embodiment block diagram of the application's distributed file system;
Fig. 7 is the embodiment block diagram of the application's client.
Embodiment
The following embodiment of the present invention provides a kind of formula document handling method, system and client based on distributed file system.
In order to make those skilled in the art person understand the technical scheme in the embodiment of the invention better; And make the above-mentioned purpose of the embodiment of the invention, feature and advantage can be more obviously understandable, below in conjunction with accompanying drawing technical scheme in the embodiment of the invention done further detailed explanation.
Distributed file system among the application embodiment comprises metadata node; The plurality of client end of communicating by letter with metadata node; And the plurality of data memory node of communicating by letter with metadata node; Wherein each data memory node is made up of at least one memory device, and the memory property of each memory device maybe be different.
Referring to Fig. 2, be the first embodiment flow chart of the application based on the document handling method of distributed file system:
Step 201: preserve consistency Hash ring in the client based on dummy node.
For the data memory node in the distributed file system; Each memory device on it corresponds at least one dummy node on the consistency Hash ring according to the size of its memory capacity; Each dummy node is preserved the device identification and the user mode of the memory device under the dummy node, according to the value that each dummy node distributes dummy node that adds up to of dummy node.Further, per two adjacent dummy nodes do not belong to same memory device, so that when write operation, reduce the possibility of choosing the dummy node that belongs to same memory device.
Step 202: client is sent file to metadata node and is write request.
When the data in the client terminal local buffer memory surpass predetermined number, need these data be write with document form in the memory device of data memory node, therefore send file and write request to metadata node.
Step 203: receive metadata node and write the file identification that request is returned for this document distribution according to file; And form the data block sign of each data block of this document, the copy amount and the selectivity factor of data block, the selectivity factor initialization value is zero.
Wherein, can represent with File inode number for the file identification of file allocation; The sign of each data block can represent that stripe id distributes according to data block order hereof with stripe id, and in distributed file system, file generally carries out piecemeal according to preset size; The copy amount of data block can be represented with replication count; Usually each data block needs to be preserved many parts; Be copy amount, this copy amount can be set by the keeper, and the different copies of common same data block are kept on the different storage device of different nodes; When guaranteeing certain memory device fault, do not influence from other storage and read this data block; Selectivity factor representes that with choice factor its initial value is 0, each data block being carried out cryptographic hash when calculating, can add an adjustment as required, so as on consistency Hash ring corresponding dummy node.
Step 204: form each data block of this document for order, the data block sign of the file identification of this document, data block and selectivity factor are carried out Hash operation obtain cryptographic hash.
Step 205: search consistency Hash ring according to cryptographic hash, obtain value, begin on consistency Hash ring, to select clockwise the dummy node consistent with copy amount from this dummy node greater than the minimum dummy node of cryptographic hash from consistency Hash ring.
Step 206: when selected dummy node belongs to different storage devices respectively, and the user mode of different storage devices is when being upstate, data block sent on the different storage devices store.
Further; The user mode that at least two dummy nodes in the dummy node of said selection belong to same data memory node or certain memory device is not upstate; Then can the value of selectivity factor be added one, return step 204 then, calculate cryptographic hash with the value of current selectivity factor; Belong to different storage devices respectively until selected dummy node, and the user mode of different storage devices is a upstate.At this moment, client can send to metadata node with the value of current selectivity factor, the value of the selectivity factor of recorded data piece is updated to the value of this current selectivity factor by metadata node.
Referring to Fig. 3 A, be the second embodiment flow chart of the application based on the document handling method of distributed file system, this embodiment shows the process that generates consistency Hash ring in the distributed file system:
Step 301: safeguard the chained list of memory device in the metadata node, the device identification of this memory device is added in this chained list according to the time backward that adds distributed file system.
In distributed file system; Comprised the plurality of data memory node, comprised some memory devices on each memory node, each memory device is assigned device identification; Safeguard the chained list of a memory device in the metadata node; Write down the equipment mark of all memory devices in this chained list, these device identifications are added in the chained list according to the time backward that equipment adds system, shown in Fig. 3 B; Chained list tail place is the memory device 1 that adds system at first, and the linked list head place is the memory device N of up-to-date adding system.
Step 302: when this distributed file system starts, and reach Preset Time after starting at interval the time, metadata node receives the storaging state information that this data memory node reports at every turn.
When the distributed file system initial start, need the storaging state information of the memory device of all data memory nodes in the system be reported metadata node; After system start-up, the state of the memory device in the system in the memory node possibly change, and therefore the storaging state information that is reported memory device according to the preset time interval by data memory node to metadata node can be set in advance.
Wherein, the storaging state information that reports of data memory node can comprise: device identification, place capacity, the equipment of forming each memory device of this memory node have been used the capacity and the possibility factor; Wherein, device identification is used for a memory device of unique identification system; Place capacity is represented the total storage capacity size of this memory device; Equipment representes that with capacity this memory device used the size of memory space; Can confirm the user mode of memory device with capacity according to this; For example, if the ratio that has accounted for total storage capacity with capacity greater than preset ratio, then the user mode of memory device is " expiring state "; If less than preset ratio, then user mode is " upstate "; The possibility factor is used to represent the performance of this memory device, can use the numeric representation between 0 to 1 usually, for example; High speed storing equipment such as solid state hard disc etc., its memory property is higher, and then the possibility factor can be set to " 1 "; Low speed storage device such as SATA hard disk etc., its memory property is lower, and then the possibility factor can be set to " 0.1 "; When carrying out write operation, can select the higher memory device of memory property according to the possibility factor, to improve writing speed.
Step 303: metadata node compares according to the device identification of memory device in the storaging state information and the device identification in the chained list.
Step 304: the device identification backward of the memory device that increases newly is added in this chained list according to comparative result.
When having increased memory device newly in the distributed system, then can newly-increased memory device backward be added in the chained list according to its order that reports that belongs to data memory node.
Step 305: this chained list of metadata node traversal generates the consistency Hash ring based on dummy node according to the place capacity of each memory device in the chained list, and be this consistency Hash ring distribution version number.
Wherein, each memory device is according at least one dummy node on the corresponding said consistency Hash ring of its memory capacity size, according to the value that each dummy node distributes dummy node that adds up to of said dummy node.
Suppose to comprise in the distributed file system three data memory nodes, be respectively NodeA, NodeB and NodeC, the memory device (Storage Device) and the capacity of each memory device that comprise in above-mentioned three memory nodes are as shown in table 1 below:
Table 1
Based on the shown data memory node of above-mentioned table 1, suppose that the maximum that hash function shines upon is 1000 on consistency Hash ring, then each memory device is mapped on this consistency Hash ring according to the size of its memory capacity form with dummy node; Corresponding at least one dummy node (Vnode) of each memory device, each dummy node is according to the same intervals assign a value, and any two adjacent virtual nodes do not belong to same memory device; Referring to Fig. 3 B, be the structural representation of the consistency Hash ring that generates according to the listed memory device of last table 1: the value that wherein Vnode1 distributed is 100, and it belongs to memory device A1; Clockwise, the value that Vnode2 distributed is 200, and it belongs to memory device A2; Until the value that Vnode10 distributed is 1000; It belongs to memory device C1, can be known by Fig. 3 C, and the capacity of memory device is big more; The quantity of dummy node that then belongs to this memory device is many more; For example, the only corresponding dummy node of memory device A1, then corresponding three dummy storage node of memory device B1.
Each dummy node on the consistency Hash ring is safeguarded an information list; Comprised the device identification of the memory device under this dummy node in this information list; The place capacity of memory device, the equipment of memory device has been used capacity, the user mode and the possibility factor.
For user mode, can be divided into upstate, expire state and down state.Wherein, Upstate and expired state and can be provided with the ratio that capacity accounts for place capacity according to equipment; The information of the memory device that down state can report according to data memory node, perhaps the state of data memory node carries out equipment in the system, such as the disk failures of certain memory device; Perhaps the data memory node at its place rolls off the production line, and the user mode that this memory device then can be set is a down state.When the user mode of dummy node when expiring state, the disk space of the memory device that it is corresponding can not distribute again, but the data on it can read; When the user mode of dummy node is a down state, then the disk space of corresponding memory device can not distribute again, and the data on it also cannot read.
Referring to Fig. 4, be the three embodiment flow chart of the application based on the document handling method of distributed file system, this embodiment shows the process that writes file in the distributed file system:
Step 401: client is sent the version number of the current consistency Hash ring of preserving to metadata node.
After the memory device in the system increased, metadata node generated new consistency Hash ring, and was that this new consistency Hash ring distributes new version number.When client write file operation in execution, needs need to abide by new consistency Hash ring to carry out, and therefore in order to know whether upgraded consistency Hash ring in the metadata node, client is sent the version number of current saved to metadata node.
Step 402: whether metadata node relatively this version number is consistent with the current version number that has preserved.
After metadata node receives the version number of client transmission, compare,, explain that then consistency Hash ring does not upgrade,, explain that then consistency Hash ring upgrades if inconsistent if consistent with the version number that self preserves.
Step 403: when inconsistent, consistency Hash ring and the version number corresponding with the current version number that has preserved are sent to client.
Step 404: preserve the consistency Hash ring based on dummy node that receives in the client, each dummy node is preserved device identification, user mode and the selectivity factor of the affiliated memory device of this dummy node.
Common each virtual information list of node maintenance that connects; Like the description among aforementioned second embodiment; The device identification of the memory device in this information list under comprising this dummy node, user mode and the selectivity factor, the place capacity that can also comprise memory device has been used capacity with the equipment of the memory device that regularly upgrades.
Step 405: client is sent file to metadata node and is write request.
Step 406: receive metadata node and write the file identification that request is returned for this document distribution according to file; And form the data block sign of each data block of this document, the copy amount and the selectivity factor of data block, the selectivity factor initialization value is zero.
Wherein, can represent with File inode number for the file identification of file allocation; The sign of each data block can represent that stripe id distributes according to data block order hereof with stripe id, and in distributed file system, file generally carries out piecemeal according to preset size; The copy amount of data block can be represented with replication count; Usually each data block needs to be preserved many parts; Be copy amount, this copy amount can be set by the keeper, and the different copies of common same data block are kept on the different storage device of different nodes; When guaranteeing certain memory device fault, do not influence from other storage and read this data block; Selectivity factor representes that with choice factor its initial value is 0, each data block being carried out cryptographic hash when calculating, can add an adjustment as required, so as on consistency Hash ring corresponding dummy node.
Step 407: form each data block of this document for order, the data block sign of the file identification of this document, data block and selectivity factor are carried out Hash operation obtain cryptographic hash.
Step 408: search consistency Hash ring according to cryptographic hash, obtain value, begin on consistency Hash ring, to select clockwise the dummy node consistent with copy amount from this dummy node greater than the minimum dummy node of cryptographic hash from consistency Hash ring.
After obtaining cryptographic hash; Can specifically adopt dichotomy on consistency Hash ring, to search value greater than the minimum dummy node of this cryptographic hash; Because the dummy node on the consistency Hash ring is arranged according to clockwise direction; Therefore begin to obtain clockwise the dummy node consistent from this minimum value corresponding virtual node,, can know memory device, user mode and the possibility factor that each dummy node is affiliated according to the information list of safeguarding on the dummy node with copy amount.
Step 409: judge whether the corresponding memory device of selected dummy node satisfies three preset conditions, if then execution in step 411; Otherwise, execution in step 410.
Wherein, three preset conditions comprise:
1, the pairing memory device of selected dummy node is all inequality; Have only the corresponding memory device of dummy node all inequality, could guarantee that the copy of data block can not be kept on the same memory device purpose of redundancy backup in the assurance system thus.
The user mode of the memory device that 2, selected dummy node is corresponding is upstate; The user mode of having only memory device is that upstate could support to write data, has expired state and down state and has not all supported to write data.
3, the selectivity factor of selected dummy node is 1 according to the random function result calculated; Because each memory device is provided with selectivity factor according to the height of its memory property; The application embodiment can adopt following random algorithm: all provide random function random in the programming language usually, utilize this function can produce the random number between 0～1, the hypothesis possibility factor is C in the present embodiment; Then C is imported above-mentioned random function random; Export a result of calculation R, if R less than C then return value 1, otherwise return value 0.
Step 410: the value of current selectivity factor is added one, return step 407.
When at least one condition in above-mentioned three conditions does not satisfy, need selectivity factor be added one, and recomputate cryptographic hash, till satisfying above-mentioned three conditions according to this new selectivity factor.
Step 411: data block sent on the corresponding different storage devices of the dummy node of selection store.
When above-mentioned three conditions all satisfy; Be that selected dummy node belongs to different storage devices respectively; And the user mode of different storage devices is a upstate; And the stochastic arithmetic value of the possibility factor of different storage devices is to be 1 o'clock, can data block be dealt on the pairing memory device of the dummy node consistent with copy amount to store.
Step 412: client sends to metadata node with the value of the current selectivity factor of the data block of composing document.
Step 413: metadata node is updated to the initialization value of the selectivity factor of data block the value of the current selectivity factor that receives.
Referring to Fig. 5, be the four embodiment flow chart of the application based on the document handling method of distributed file system, this embodiment shows the process that reads file in the distributed file system:
Step 501: client is sent file to metadata node and is read request, and this document reads include file sign in the request.
Step 502: metadata node is obtained the data block sign of each data block of forming this document, the copy amount of data block and the value of selectivity factor according to file identification.
Step 503: metadata node sends to client with the data block sign and the selectivity factor of file identification, each data block.
Step 504: for each data block of forming this document in proper order, client is according to file identification, and the value of the data block of each data block sign and selectivity factor is carried out Hash operation and obtained cryptographic hash.
Wherein, the Hash operation that the Hash operation that client adopted is adopted when writing data is consistent, to guarantee that according to file identification the cryptographic hash that the value of the data block of each data block sign and selectivity factor calculates is constant.
Step 505: client is searched consistency Hash ring according to this cryptographic hash; Obtain value from consistency Hash ring, begin on consistency Hash ring, to select clockwise the dummy node consistent with copy amount from this dummy node greater than the minimum dummy node of this cryptographic hash.
Step 506: client is according to the dummy node that finds, read block to the corresponding memory device of one of them dummy node.
For each data block; When carrying out write operation; All preserve, so after finding the memory device consistent, can therefrom select a memory device read block to get final product with copy amount through the memory device consistent with copy amount; For example, can select greater than the corresponding memory device of the value of the minimum dummy node of cryptographic hash.
Step 507: client is formed said file according to the data block sign of the data block that reads with all data blocks in order.
Visible by the foregoing description; Owing to preserved consistency Hash ring in the client; Therefore when execution writes file operation, need not metadata node and preserve the distribution situation of file on the memory device of each memory node through the file layout allocation table in advance, file allocation and the positioning action originally on metadata node, carried out are moved to client executing; Metadata node only needs the selectivity factor of transaction file data block, has therefore reduced taking the metadata node memory space; Because each client only need be when self carries out write operation be carried out the location of memory device according to consistency Hash ring; Rather than position by the request that writes of metadata node to all clients in the system, therefore can improve the locating speed when writing data; And, thereby through the user mode of the corresponding memory device of each dummy node on the consistency Hash ring being judged self adaptation carries out the selection of memory device, realized thus making system reach load balancing according to the residual capacity of disk.
Corresponding based on the embodiment of the document handling method of distributed file system with the application, the application also provides the embodiment of distributed file system and client.
Referring to Fig. 6, be the embodiment block diagram of the application's distributed file system:
This distributed file system comprises: metadata node 610, with client 620 and data memory node 630 that metadata node 610 is communicated by letter respectively, each memory node 630 is made up of at least one memory device 631.Convenient for example, only show a client 610 and three memory nodes 630 among Fig. 6, and show a memory device, two memory devices and three memory devices in each memory node respectively.Need to prove that in actual application, the quantity of the memory device on client, data memory node and each data memory node can be provided with as required, the setting that the embodiment among Fig. 6 is not used in system limits.
Wherein, Said client 620; Be used to preserve consistency Hash ring based on dummy node; Wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity, and each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node, according to the value that each dummy node distributes dummy node that adds up to of said dummy node; And send file to metadata node and write request;
Said metadata node 610; Be used for writing request and be the file identification of said file allocation to what said client was returned according to said file; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Said memory device 631 is used for the data block that said client is sent is preserved.
Further, said metadata node 610 also is used to safeguard the chained list of memory device, and the device identification of said memory device is added in the said chained list according to the time backward that adds distributed file system;
Said data memory node 630; Be used for when said distributed file system starts; And reach Preset Time after starting at interval the time at every turn; Report the storaging state information of said data memory node, said storaging state information to comprise that device identification, place capacity, the equipment of each memory device of forming said memory node used the capacity and the possibility factor to said metadata node;
Said metadata node 610 also is used for comparing according to the device identification of storaging state information memory device and the device identification in the chained list, and the device identification backward of the memory device that increases newly is added in the said chained list; Travel through said chained list, and generate said consistency Hash ring, and be that said consistency Hash ring distributes version number based on dummy node according to the place capacity of said each memory device; Said consistency Hash ring and said version number are sent to said client.
Further, said client 620 also was used for before metadata node transmission file writes request the version number that sends the current consistency Hash ring of preserving to said metadata node;
Said metadata node 610, whether with the current version number that preserved consistent, when inconsistent, consistency Hash ring and the version number corresponding with the said current version number that has preserved are sent to said client if also being used for more said version number.
Further; Said client 620; Also be used for also having preserved the possibility factor of the memory device under the said dummy node, then belong to different storage devices respectively, and the user mode of said memory device is when being upstate at dummy node when said selection when each dummy node; Whether the stochastic arithmetic value of judging the possibility factor of said memory device is 1, if be 1 execution data block sent on the said different storage devices store.
Further; Said client 620; Also be used for user mode that at least two dummy nodes when the dummy node of said selection belong to same data memory node or said memory device for the stochastic arithmetic value of the possibility factor of upstate or said memory device is not 1, then the value with said selectivity factor adds one; Returning execution carries out Hash operation to the data block of the file identification of said file, said data block sign and selectivity factor and obtains cryptographic hash; Dummy node until said selection belongs to different storage devices respectively; And the user mode of said different storage devices is a upstate, and the stochastic arithmetic value of the said possibility factor is 1.
Further, said client 620, also be used for when said data block is sent to store on the said different storage devices after, the value of current selectivity factor is sent to said metadata node;
Said metadata node 610 also is used for the initialization value of the selectivity factor of said data block is updated to the value of said current selectivity factor.
Further, said client 620 also is used for sending file to said metadata node and reads request, and said file reads include file sign in the request;
Said metadata node 610 also is used for obtaining the data block sign of each data block of forming said file, the copy amount of data block and the value of selectivity factor according to said file identification; The data block sign and the selectivity factor of said file identification, each data block are sent to client;
Referring to Fig. 7, be the embodiment block diagram of the application's client:
This client application is in the distributed file system shown in the application embodiment; The framework of this system can be referring to system embodiment shown in Figure 6; Comprise metadata node; Said metadata node is communicated by letter with data memory node with said client respectively, and each back end is made up of at least one memory device.This client comprises: preserve unit 710, transmitting element 720, receiving element 730, arithmetic element 740 and acquiring unit 750.
Wherein, Preserve unit 710; Be used to preserve consistency Hash ring based on dummy node; Wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity, and each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node, according to the value that each dummy node distributes dummy node that adds up to of said dummy node;
Transmitting element 720 is used for sending file to metadata node and writes request;
Receiving element 730; Be used to receive metadata node and be the file identification of said file allocation according to what said file write that request returns; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Acquiring unit 750; Be used for searching said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
Transmitting element 720 also is used for belonging to different storage devices respectively when selected dummy node, and the user mode of said different storage devices is when being upstate, data block is sent on the said different storage devices store.
Further, said receiving element 730 also is used for when said distributed file system starts, and reaches Preset Time after starting at interval the time at every turn, receives the consistency Hash ring that said metadata node sends and the version number of said consistency Hash ring; Wherein, Said metadata node be used to safeguard memory device chained list; The device identification of said memory device is added in the said chained list according to the time backward that adds distributed file system; When said distributed file system starts; Receive the storaging state information that said data memory node reports, said storaging state information comprises that device identification, place capacity, the equipment of each memory device of forming said memory node used the capacity and the possibility factor, compares according to the device identification of memory device in the storaging state information and the device identification in the chained list; The device identification backward of the memory device that increases newly is added in the said chained list; Travel through said chained list, and generate said consistency Hash ring, and be that said consistency Hash ring distributes version number based on dummy node according to the place capacity of said each memory device;
Said transmitting element 720 also was used for before metadata node transmission file writes request the version number that sends the current consistency Hash ring of preserving to said metadata node;
Said receiving element 730 also is used in the more said version number of said metadata node and the current version number that has preserved when inconsistent, receive that said metadata node sends with corresponding consistency Hash ring and version number of the said current version number that has preserved.
Further; This client can also comprise (not shown among Fig. 7): judging unit; The possibility factor that is used for the memory device under said each dummy node is also preserved said dummy node; And when the dummy node of said selection belongs to different storage devices respectively, and the user mode of said memory device judges whether the stochastic arithmetic value of the possibility factor of said memory device is 1 when being upstate;
Said transmitting element 720 also is used for when the said stochastic arithmetic value of said judgment unit judges is 1, data block is sent on the said different storage devices store.
Further; Said arithmetic element 740; Also being used for user mode that at least two dummy nodes when the dummy node of said selection belong to same data memory node or said memory device is not 1 for the stochastic arithmetic value of the possibility factor of upstate or said memory device; Then the value with said selectivity factor adds one; And return execution and the data block of the file identification of said file, said data block sign and selectivity factor are carried out Hash operation obtain cryptographic hash; Dummy node until said selection belongs to different storage devices respectively, and the user mode of said different storage devices is upstate, and the stochastic arithmetic value of the said possibility factor is 1.
Further; Said transmitting element 720; Also be used for when said transmitting element said data block is sent to store on the said different storage devices after; The value of current selectivity factor is sent to said metadata node, so that said metadata node is updated to the initialization value of the selectivity factor of said data block the value of said current selectivity factor.
Further, said transmitting element 720 also is used for sending file to said metadata node and reads request, and said file reads include file sign in the request;
Said receiving element 730 also is used to receive data block sign, the copy amount of data block and the value of selectivity factor of each data block of the said file of composition that said metadata node sends obtains according to said file identification;
Said arithmetic element 740 also is used for for each data block of forming said file in proper order, and client is according to said file identification, and the value of the data block of each data block sign and selectivity factor is carried out Hash operation and obtained cryptographic hash;
Said acquiring unit 750; Also be used for searching said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
Said client also comprises (not shown among Fig. 7): reading unit; According to the dummy node that finds; To the corresponding memory device of one of them dummy node, read said data block, and all data blocks are formed said file in order according to the data block sign of the data block that reads.
Description through to above execution mode can be known; Send file by client to metadata node among the application embodiment and write request; Receive metadata node and write the file identification that request is returned for this document distribution according to file; And copy amount and the initialization value of forming data block sign, the data block of each data block of said file be zero selectivity factor, forms each data block of said file for order, the data block sign of the file identification of file, data block and selectivity factor carried out Hash operation obtain cryptographic hash; Search the consistency Hash ring of preserving on the client according to cryptographic hash; Obtain value, begin on consistency Hash ring, to select the dummy node consistent clockwise from this dummy node, when selected dummy node belongs to different storage devices respectively with copy amount greater than the minimum dummy node of this cryptographic hash; And when the user mode of different storage devices is upstate, data block is sent on the different storage devices store.In the distributed file system of the application embodiment; Owing to preserved consistency Hash ring in the client; Therefore when execution writes file operation; Need not metadata node and preserve the distribution situation of file on the memory device of each memory node through the file layout allocation table in advance; File allocation and the positioning action originally on metadata node, carried out are moved to client executing, and metadata node only needs the selectivity factor of transaction file data block, has therefore reduced taking the metadata node memory space; Because each client only need be when self carries out write operation be carried out the location of memory device according to consistency Hash ring; Rather than position by the request that writes of metadata node to all clients in the system, therefore can improve the locating speed when writing data; And, thereby through the user mode of the corresponding memory device of each dummy node on the consistency Hash ring being judged self adaptation carries out the selection of memory device, realized thus making system reach load balancing according to the residual capacity of disk.
The technology that those skilled in the art can be well understood in the embodiment of the invention can realize by the mode that software adds essential general hardware platform.Based on such understanding; The part that technical scheme in the embodiment of the invention contributes to prior art in essence in other words can be come out with the embodied of software product; This computer software product can be stored in the storage medium, like ROM/RAM, magnetic disc, CD etc., comprises that some instructions are with so that a computer equipment (can be a personal computer; Server, the perhaps network equipment etc.) carry out the described method of some part of each embodiment of the present invention or embodiment.
Each embodiment in this specification all adopts the mode of going forward one by one to describe, and identical similar part is mutually referring to getting final product between each embodiment, and each embodiment stresses all is the difference with other embodiment.Especially, for system embodiment, because it is basically similar in appearance to method embodiment, so description is fairly simple, relevant part gets final product referring to the part explanation of method embodiment.
Above-described embodiment of the present invention does not constitute the qualification to protection range of the present invention.Any modification of within spirit of the present invention and principle, being done, be equal to replacement and improvement etc., all should be included within protection scope of the present invention.
Claims (20)
1. document handling method based on distributed file system; Said system comprises metadata node and the client and the data memory node of communicating by letter respectively with said metadata node; Each data memory node is made up of at least one memory device; It is characterized in that; Preserve the consistency Hash ring based on dummy node in the client, wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity, and each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node; According to the value that each dummy node distributes dummy node that adds up to of said dummy node, said method comprises:
Client is sent file to metadata node and is write request;
Receive metadata node and be the file identification of said file allocation according to what said file write that request returns; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Form each data block of said file for order, the data block sign of the file identification of said file, said data block and selectivity factor are carried out Hash operation obtain cryptographic hash;
Search said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
When selected dummy node belongs to different storage devices respectively, and the user mode of said different storage devices is when being upstate, data block sent on the said different storage devices store.
2. method according to claim 1 is characterized in that, also comprises:
Safeguard the chained list of memory device in the metadata node, the device identification of said memory device is added in the said chained list according to the time backward that adds distributed file system;
When said distributed file system starts; And reach Preset Time after starting at interval the time at every turn; Metadata node receives the storaging state information that said data memory node reports, and said storaging state information comprises that device identification, place capacity, the equipment of each memory device of forming said memory node used the capacity and the possibility factor;
Metadata node compares according to the device identification of memory device in the storaging state information and the device identification in the chained list, and the device identification backward of the memory device that increases newly is added in the said chained list;
Metadata node travels through said chained list, and generates said consistency Hash ring based on dummy node according to the place capacity of said each memory device, and is that said consistency Hash ring distributes version number.
3. method according to claim 2 is characterized in that, said client also comprised before metadata node transmission file writes request:
Client is sent the version number of the current consistency Hash ring of preserving to said metadata node;
Whether the more said version number of said metadata node is consistent with the current version number that has preserved, when inconsistent, consistency Hash ring and the version number corresponding with the said current version number that has preserved sent to said client.
4. method according to claim 2 is characterized in that, said each dummy node is also preserved the possibility factor of the memory device under the said dummy node;
When the dummy node of said selection belongs to different storage devices respectively, and the user mode of said memory device is when being upstate, and said method also comprises:
Whether the stochastic arithmetic value of judging the possibility factor of said memory device is 1, if be 1 execution data block sent on the said different storage devices store.
5. method according to claim 4 is characterized in that, also comprises:
The user mode that at least two dummy nodes in the dummy node of said selection belong to same data memory node or said memory device is not 1 for the stochastic arithmetic value of the possibility factor of upstate or said memory device, and then the value with said selectivity factor adds one;
Returning execution carries out Hash operation to the data block of the file identification of said file, said data block sign and selectivity factor and obtains cryptographic hash; Dummy node until said selection belongs to different storage devices respectively; And the user mode of said different storage devices is a upstate, and the stochastic arithmetic value of the said possibility factor is 1.
6. method according to claim 1 is characterized in that, also comprises:
When said data block is sent to store on the said different storage devices after, client sends to said metadata node with the value of current selectivity factor;
Said metadata node is updated to the initialization value of the selectivity factor of said data block the value of said current selectivity factor.
7. method according to claim 6 is characterized in that, also comprises:
Client is sent file to said metadata node and is read request, and said file reads include file sign in the request;
Metadata node is obtained the data block sign of each data block of forming said file, the copy amount of data block and the value of selectivity factor according to said file identification;
Metadata node sends to client with the data block sign and the selectivity factor of said file identification, each data block;
For each data block of forming said file in proper order, client is according to said file identification, and the value of the data block of each data block sign and selectivity factor is carried out Hash operation and obtained cryptographic hash;
Client is searched said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
Client reads said data block according to the dummy node that finds to the corresponding memory device of one of them dummy node;
Client is formed said file according to the data block sign of the data block that reads with all data blocks in order.
8. distributed file system comprises: metadata node and the client and the data memory node of communicating by letter respectively with said metadata node, and each data memory node is made up of at least one memory device, it is characterized in that,
Said client; Be used to preserve consistency Hash ring based on dummy node; Wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity; Each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node, according to the value that each dummy node distributes dummy node that adds up to of said dummy node; And send file to metadata node and write request;
Said metadata node; Be used for writing request and be the file identification of said file allocation to what said client was returned according to said file; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Said client also is used for forming each data block of said file for order, the data block sign of the file identification of said file, said data block and selectivity factor is carried out Hash operation obtain cryptographic hash; Search said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash; When selected dummy node belongs to different storage devices respectively, and the user mode of said different storage devices sends to data block on the said different storage devices when being upstate;
Said memory device is used for the data block that said client is sent is preserved.
9. system according to claim 8 is characterized in that,
Said metadata node also is used to safeguard the chained list of memory device, and the device identification of said memory device is added in the said chained list according to the time backward that adds distributed file system;
Said data memory node; Be used for when said distributed file system starts; And reach Preset Time after starting at interval the time at every turn; Report the storaging state information of said data memory node, said storaging state information to comprise that device identification, place capacity, the equipment of each memory device of forming said memory node used the capacity and the possibility factor to said metadata node;
Said metadata node also is used for comparing according to the device identification of storaging state information memory device and the device identification in the chained list, and the device identification backward of the memory device that increases newly is added in the said chained list; Travel through said chained list, and generate said consistency Hash ring, and be that said consistency Hash ring distributes version number based on dummy node according to the place capacity of said each memory device; Said consistency Hash ring and said version number are sent to said client.
10. system according to claim 9 is characterized in that,
Said client also was used for before metadata node transmission file writes request the version number that sends the current consistency Hash ring of preserving to said metadata node;
Said metadata node, whether with the current version number that preserved consistent, when inconsistent, consistency Hash ring and the version number corresponding with the said current version number that has preserved are sent to said client if also being used for more said version number.
11. system according to claim 9 is characterized in that,
Said client; The possibility factor that also is used for the memory device under each dummy node has also been preserved said dummy node; Then belong to different storage devices respectively at dummy node when said selection; And when the user mode of said memory device is upstate, judge whether the stochastic arithmetic value of the possibility factor of said memory device is 1, if be 1 execution data block sent on the said different storage devices store.
12. system according to claim 11 is characterized in that,
Said client; Also be used for user mode that at least two dummy nodes when the dummy node of said selection belong to same data memory node or said memory device for the stochastic arithmetic value of the possibility factor of upstate or said memory device is not 1, then the value with said selectivity factor adds one; Returning execution carries out Hash operation to the data block of the file identification of said file, said data block sign and selectivity factor and obtains cryptographic hash; Dummy node until said selection belongs to different storage devices respectively; And the user mode of said different storage devices is a upstate, and the stochastic arithmetic value of the said possibility factor is 1.
13. system according to claim 8 is characterized in that,
Said client, also be used for when said data block is sent to store on the said different storage devices after, the value of current selectivity factor is sent to said metadata node;
Said metadata node also is used for the initialization value of the selectivity factor of said data block is updated to the value of said current selectivity factor.
14. system according to claim 13 is characterized in that,
Said client also is used for sending file to said metadata node and reads request, and said file reads include file sign in the request;
Said metadata node also is used for obtaining the data block sign of each data block of forming said file, the copy amount of data block and the value of selectivity factor according to said file identification; The data block sign and the selectivity factor of said file identification, each data block are sent to client;
Said client also is used for for each data block of forming said file in proper order, and according to said file identification, the value of the data block of each data block sign and selectivity factor is carried out Hash operation and obtained cryptographic hash; Search said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash; According to the dummy node that finds, to the corresponding memory device of one of them dummy node, read said data block; Data block sign according to the data block that reads is formed said file in order with all data blocks.
15. a client is applied in the distributed file system, said system comprises metadata node; Said metadata node is communicated by letter with data memory node with said client respectively; Each back end is made up of at least one memory device, it is characterized in that, said client comprises:
Preserve the unit; Be used to preserve consistency Hash ring based on dummy node; Wherein each memory device corresponds at least one dummy node on the said consistency Hash ring according to the size of its memory capacity; Each dummy node is preserved the device identification and the user mode of the memory device under the said dummy node, according to the value that each dummy node distributes dummy node that adds up to of said dummy node;
Transmitting element is used for sending file to metadata node and writes request;
Receiving element; Be used to receive metadata node and be the file identification of said file allocation according to what said file write that request returns; And form the data block sign of each data block of said file, the copy amount and the selectivity factor of data block, said selectivity factor initialization value is zero;
Arithmetic element is used for forming each data block of said file for order, the data block sign of the file identification of said file, said data block and selectivity factor is carried out Hash operation obtain cryptographic hash;
Acquiring unit; Be used for searching said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
Transmitting element also is used for belonging to different storage devices respectively when selected dummy node, and the user mode of said different storage devices is when being upstate, data block is sent on the said different storage devices store.
16. client according to claim 15 is characterized in that,
Said receiving element also is used for when said distributed file system starts, and reaches Preset Time after starting at interval the time at every turn, receives the consistency Hash ring that said metadata node sends and the version number of said consistency Hash ring; Wherein, Said metadata node be used to safeguard memory device chained list; The device identification of said memory device is added in the said chained list according to the time backward that adds distributed file system; When said distributed file system starts; Receive the storaging state information that said data memory node reports, said storaging state information comprises that device identification, place capacity, the equipment of each memory device of forming said memory node used the capacity and the possibility factor, compares according to the device identification of memory device in the storaging state information and the device identification in the chained list; The device identification backward of the memory device that increases newly is added in the said chained list; Travel through said chained list, and generate said consistency Hash ring, and be that said consistency Hash ring distributes version number based on dummy node according to the place capacity of said each memory device;
Said transmitting element also was used for before metadata node transmission file writes request the version number that sends the current consistency Hash ring of preserving to said metadata node;
Said receiving element also is used in the more said version number of said metadata node and the current version number that has preserved when inconsistent, receive that said metadata node sends with corresponding consistency Hash ring and version number of the said current version number that has preserved.
17. client according to claim 16 is characterized in that, also comprises:
Judging unit; The possibility factor that is used for the memory device under said each dummy node is also preserved said dummy node; And when the dummy node of said selection belongs to different storage devices respectively; And when the user mode of said memory device is upstate, judge whether the stochastic arithmetic value of the possibility factor of said memory device is 1;
Said transmitting element also is used for when the said stochastic arithmetic value of said judgment unit judges is 1, data block is sent on the said different storage devices store.
18. client according to claim 17 is characterized in that,
Said arithmetic element; Also being used for user mode that at least two dummy nodes when the dummy node of said selection belong to same data memory node or said memory device is not 1 for the stochastic arithmetic value of the possibility factor of upstate or said memory device; Then the value with said selectivity factor adds one; And return execution and the data block of the file identification of said file, said data block sign and selectivity factor are carried out Hash operation obtain cryptographic hash; Dummy node until said selection belongs to different storage devices respectively; And the user mode of said different storage devices is a upstate, and the stochastic arithmetic value of the said possibility factor is 1.
19. client according to claim 1 is characterized in that,
Said transmitting element; Also be used for when said transmitting element said data block is sent to store on the said different storage devices after; The value of current selectivity factor is sent to said metadata node, so that said metadata node is updated to the initialization value of the selectivity factor of said data block the value of said current selectivity factor.
20. client according to claim 19 is characterized in that,
Said transmitting element also is used for sending file to said metadata node and reads request, and said file reads include file sign in the request;
Said receiving element also is used to receive data block sign, the copy amount of data block and the value of selectivity factor of each data block of the said file of composition that said metadata node sends obtains according to said file identification;
Said arithmetic element also is used for for each data block of forming said file in proper order, and client is according to said file identification, and the value of the data block of each data block sign and selectivity factor is carried out Hash operation and obtained cryptographic hash;
Said acquiring unit; Also be used for searching said consistency Hash ring according to said cryptographic hash; Obtain value from said consistency Hash ring, begin on said consistency Hash ring, to select clockwise the dummy node consistent with said copy amount from said dummy node greater than the minimum dummy node of said cryptographic hash;
Said client also comprises:
Reading unit according to the dummy node that finds, reads said data block to the corresponding memory device of one of them dummy node, and according to the data block sign of the data block that reads all data blocks is formed said file in order.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110172216 CN102843403A (en) | 2011-06-23 | 2011-06-23 | File processing method based on distributed file system, system, and client |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110172216 CN102843403A (en) | 2011-06-23 | 2011-06-23 | File processing method based on distributed file system, system, and client |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102843403A true CN102843403A (en) | 2012-12-26 |
Family
ID=47370462
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110172216 Pending CN102843403A (en) | 2011-06-23 | 2011-06-23 | File processing method based on distributed file system, system, and client |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102843403A (en) |
Cited By (36)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103384550A (en) * | 2012-12-28 | 2013-11-06 | 华为技术有限公司 | Data storage method and device |
| CN103441906A (en) * | 2013-09-25 | 2013-12-11 | 哈尔滨工业大学 | System for detecting abnormity of proxy cache cluster based on automatic computing |
| CN104158897A (en) * | 2014-08-25 | 2014-11-19 | 曙光信息产业股份有限公司 | Updating method of file layout in distributed file system |
| CN104636286A (en) * | 2015-02-06 | 2015-05-20 | 华为技术有限公司 | Data access method and equipment |
| CN104780201A (en) * | 2015-03-02 | 2015-07-15 | 新浪网技术(中国)有限公司 | Data packet processing method and device for use in IPVS (Internet Protocol Virtual Server) cluster |
| CN105278882A (en) * | 2015-10-26 | 2016-01-27 | 创新科存储技术有限公司 | Disk management method of distributed file system |
| WO2016029744A1 (en) * | 2014-08-27 | 2016-03-03 | 华为技术有限公司 | Metadata recovery method and relevant device |
| CN105653629A (en) * | 2015-12-28 | 2016-06-08 | 湖南蚁坊软件有限公司 | Hash ring-based distributed data filter method |
| CN106034144A (en) * | 2015-03-12 | 2016-10-19 | 中国人民解放军国防科学技术大学 | Load-balancing-based virtual asset data storage method |
| CN106445409A (en) * | 2016-09-13 | 2017-02-22 | 郑州云海信息技术有限公司 | Distributed block storage data writing method and device |
| CN106776000A (en) * | 2016-11-17 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of distributed file system Implementing Load Balance |
| CN106897360A (en) * | 2017-01-09 | 2017-06-27 | 深圳市中博科创信息技术有限公司 | Document handling method and device based on distributed file system |
| CN107071031A (en) * | 2017-04-19 | 2017-08-18 | 电子科技大学 | Distributed block memory system data based on chunk blocks version number recovers decision method |
| CN107908372A (en) * | 2017-12-14 | 2018-04-13 | 南京云创大数据科技股份有限公司 | A kind of more fusion storage methods of strange land isomery |
| CN108132830A (en) * | 2016-12-01 | 2018-06-08 | 北京金山云网络技术有限公司 | A kind of method for scheduling task, apparatus and system |
| CN108197323A (en) * | 2018-02-05 | 2018-06-22 | 丁武轩 | Applied to distributed system map data processing method |
| CN108304555A (en) * | 2018-02-05 | 2018-07-20 | 丁武轩 | Distributed maps data processing method |
| CN104754000B (en) * | 2013-12-30 | 2018-08-24 | 国家电网公司 | A kind of load-balancing method and system |
| CN109739435A (en) * | 2018-12-03 | 2019-05-10 | 中科恒运股份有限公司 | File storage and update method and device |
| CN109885423A (en) * | 2019-01-04 | 2019-06-14 | 上海七印信息科技有限公司 | A kind of block chain source data backup method |
| CN110046160A (en) * | 2019-03-15 | 2019-07-23 | 中国科学院计算技术研究所 | A kind of consistency Hash storage system construction method based on band |
| CN110086840A (en) * | 2018-01-26 | 2019-08-02 | 浙江宇视科技有限公司 | Image data recording method, device and computer readable storage medium |
| CN110096227A (en) * | 2019-03-28 | 2019-08-06 | 北京奇艺世纪科技有限公司 | Date storage method, data processing method, device, electronic equipment and computer-readable medium |
| CN110427384A (en) * | 2019-07-30 | 2019-11-08 | 北京明略软件系统有限公司 | The sending method and device of prompt information, storage medium, electronic device |
| CN110457263A (en) * | 2019-08-13 | 2019-11-15 | 北京首都在线科技股份有限公司 | A kind of date storage method and device |
| CN110603517A (en) * | 2017-06-08 | 2019-12-20 | 日立数据管理有限公司 | De-duplicating distributed erasure-coded objects |
| CN110633053A (en) * | 2019-09-16 | 2019-12-31 | 北京马赫谷科技有限公司 | Storage capacity balancing method, object storage method and device |
| CN111552441A (en) * | 2020-04-29 | 2020-08-18 | 重庆紫光华山智安科技有限公司 | Data storage method and device, main node and distributed system |
| CN112230861A (en) * | 2020-10-26 | 2021-01-15 | 金钱猫科技股份有限公司 | Data storage method and terminal based on consistent hash algorithm |
| CN112241336A (en) * | 2019-07-19 | 2021-01-19 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for backing up data |
| CN112948120A (en) * | 2021-03-18 | 2021-06-11 | 广东好太太智能家居有限公司 | Load balancing method, system, device and storage medium |
| CN112988680A (en) * | 2021-03-30 | 2021-06-18 | 联想凌拓科技有限公司 | Data acceleration method, cache unit, electronic device and storage medium |
| CN113282243A (en) * | 2021-06-09 | 2021-08-20 | 杭州海康威视系统技术有限公司 | Method and device for storing object file |
| CN113672524A (en) * | 2021-08-20 | 2021-11-19 | 上海哔哩哔哩科技有限公司 | Data processing method and system based on multi-level cache |
| CN113806298A (en) * | 2021-08-09 | 2021-12-17 | 中国地质大学（武汉） | Distributed file system storage optimization method, computer equipment and storage medium |
| CN116522395A (en) * | 2023-05-22 | 2023-08-01 | 上海汉朔信息科技有限公司 | Method and system for processing data files in big data storage system |
-
2011
- 2011-06-23 CN CN 201110172216 patent/CN102843403A/en active Pending
Cited By (50)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103384550B (en) * | 2012-12-28 | 2016-05-25 | 华为技术有限公司 | The method of storage data and device |
| CN103384550A (en) * | 2012-12-28 | 2013-11-06 | 华为技术有限公司 | Data storage method and device |
| CN103441906A (en) * | 2013-09-25 | 2013-12-11 | 哈尔滨工业大学 | System for detecting abnormity of proxy cache cluster based on automatic computing |
| CN104754000B (en) * | 2013-12-30 | 2018-08-24 | 国家电网公司 | A kind of load-balancing method and system |
| CN104158897B (en) * | 2014-08-25 | 2017-06-23 | 曙光信息产业股份有限公司 | The update method of file layout in a kind of distributed file system |
| CN104158897A (en) * | 2014-08-25 | 2014-11-19 | 曙光信息产业股份有限公司 | Updating method of file layout in distributed file system |
| WO2016029744A1 (en) * | 2014-08-27 | 2016-03-03 | 华为技术有限公司 | Metadata recovery method and relevant device |
| CN104636286A (en) * | 2015-02-06 | 2015-05-20 | 华为技术有限公司 | Data access method and equipment |
| CN104636286B (en) * | 2015-02-06 | 2018-07-03 | 华为技术有限公司 | For the method and apparatus of data access |
| CN104780201A (en) * | 2015-03-02 | 2015-07-15 | 新浪网技术(中国)有限公司 | Data packet processing method and device for use in IPVS (Internet Protocol Virtual Server) cluster |
| CN106034144A (en) * | 2015-03-12 | 2016-10-19 | 中国人民解放军国防科学技术大学 | Load-balancing-based virtual asset data storage method |
| CN105278882A (en) * | 2015-10-26 | 2016-01-27 | 创新科存储技术有限公司 | Disk management method of distributed file system |
| CN105653629B (en) * | 2015-12-28 | 2020-03-13 | 湖南蚁坊软件股份有限公司 | Distributed data filtering method based on Hash ring |
| CN105653629A (en) * | 2015-12-28 | 2016-06-08 | 湖南蚁坊软件有限公司 | Hash ring-based distributed data filter method |
| CN106445409A (en) * | 2016-09-13 | 2017-02-22 | 郑州云海信息技术有限公司 | Distributed block storage data writing method and device |
| CN106776000A (en) * | 2016-11-17 | 2017-05-31 | 郑州云海信息技术有限公司 | A kind of distributed file system Implementing Load Balance |
| CN108132830A (en) * | 2016-12-01 | 2018-06-08 | 北京金山云网络技术有限公司 | A kind of method for scheduling task, apparatus and system |
| CN108132830B (en) * | 2016-12-01 | 2020-12-25 | 北京金山云网络技术有限公司 | Task scheduling method, device and system |
| CN106897360A (en) * | 2017-01-09 | 2017-06-27 | 深圳市中博科创信息技术有限公司 | Document handling method and device based on distributed file system |
| CN107071031B (en) * | 2017-04-19 | 2019-11-05 | 电子科技大学 | Distributed block memory system data based on chunk block version number restores determination method |
| CN107071031A (en) * | 2017-04-19 | 2017-08-18 | 电子科技大学 | Distributed block memory system data based on chunk blocks version number recovers decision method |
| CN110603517A (en) * | 2017-06-08 | 2019-12-20 | 日立数据管理有限公司 | De-duplicating distributed erasure-coded objects |
| CN107908372A (en) * | 2017-12-14 | 2018-04-13 | 南京云创大数据科技股份有限公司 | A kind of more fusion storage methods of strange land isomery |
| CN107908372B (en) * | 2017-12-14 | 2021-09-10 | 南京云创大数据科技股份有限公司 | Heterogeneous multi-fusion storage method in different places |
| CN110086840A (en) * | 2018-01-26 | 2019-08-02 | 浙江宇视科技有限公司 | Image data recording method, device and computer readable storage medium |
| CN108197323A (en) * | 2018-02-05 | 2018-06-22 | 丁武轩 | Applied to distributed system map data processing method |
| CN108197323B (en) * | 2018-02-05 | 2021-03-30 | 深圳市易图资讯股份有限公司 | Map data processing method applied to distributed system |
| CN108304555A (en) * | 2018-02-05 | 2018-07-20 | 丁武轩 | Distributed maps data processing method |
| CN109739435A (en) * | 2018-12-03 | 2019-05-10 | 中科恒运股份有限公司 | File storage and update method and device |
| CN109739435B (en) * | 2018-12-03 | 2023-10-13 | 中科恒运股份有限公司 | File storage and updating method and device |
| CN109885423A (en) * | 2019-01-04 | 2019-06-14 | 上海七印信息科技有限公司 | A kind of block chain source data backup method |
| CN109885423B (en) * | 2019-01-04 | 2023-05-23 | 上海七印信息科技有限公司 | Block chain source data backup method |
| CN110046160A (en) * | 2019-03-15 | 2019-07-23 | 中国科学院计算技术研究所 | A kind of consistency Hash storage system construction method based on band |
| CN110096227A (en) * | 2019-03-28 | 2019-08-06 | 北京奇艺世纪科技有限公司 | Date storage method, data processing method, device, electronic equipment and computer-readable medium |
| CN112241336A (en) * | 2019-07-19 | 2021-01-19 | 伊姆西Ip控股有限责任公司 | Method, apparatus and computer program product for backing up data |
| CN110427384A (en) * | 2019-07-30 | 2019-11-08 | 北京明略软件系统有限公司 | The sending method and device of prompt information, storage medium, electronic device |
| CN110457263A (en) * | 2019-08-13 | 2019-11-15 | 北京首都在线科技股份有限公司 | A kind of date storage method and device |
| CN110457263B (en) * | 2019-08-13 | 2021-10-26 | 北京首都在线科技股份有限公司 | Data storage method and device |
| CN110633053A (en) * | 2019-09-16 | 2019-12-31 | 北京马赫谷科技有限公司 | Storage capacity balancing method, object storage method and device |
| CN110633053B (en) * | 2019-09-16 | 2020-07-10 | 北京马赫谷科技有限公司 | Storage capacity balancing method, object storage method and device |
| CN111552441A (en) * | 2020-04-29 | 2020-08-18 | 重庆紫光华山智安科技有限公司 | Data storage method and device, main node and distributed system |
| CN111552441B (en) * | 2020-04-29 | 2023-02-28 | 重庆紫光华山智安科技有限公司 | Data storage method and device, main node and distributed system |
| CN112230861A (en) * | 2020-10-26 | 2021-01-15 | 金钱猫科技股份有限公司 | Data storage method and terminal based on consistent hash algorithm |
| CN112948120A (en) * | 2021-03-18 | 2021-06-11 | 广东好太太智能家居有限公司 | Load balancing method, system, device and storage medium |
| CN112988680A (en) * | 2021-03-30 | 2021-06-18 | 联想凌拓科技有限公司 | Data acceleration method, cache unit, electronic device and storage medium |
| CN113282243A (en) * | 2021-06-09 | 2021-08-20 | 杭州海康威视系统技术有限公司 | Method and device for storing object file |
| CN113806298A (en) * | 2021-08-09 | 2021-12-17 | 中国地质大学（武汉） | Distributed file system storage optimization method, computer equipment and storage medium |
| CN113672524A (en) * | 2021-08-20 | 2021-11-19 | 上海哔哩哔哩科技有限公司 | Data processing method and system based on multi-level cache |
| CN116522395A (en) * | 2023-05-22 | 2023-08-01 | 上海汉朔信息科技有限公司 | Method and system for processing data files in big data storage system |
| CN116522395B (en) * | 2023-05-22 | 2023-10-13 | 上海汉朔信息科技有限公司 | Method and system for processing data files in big data storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102843403A (en) | File processing method based on distributed file system, system, and client | |
| AU2013409624B2 (en) | Data processing method and device in distributed file storage system | |
| US20180253484A1 (en) | Computer Implemented Method for Dynamic Sharding | |
| CN104794123B (en) | A kind of method and device building NoSQL database indexes for semi-structured data | |
| CN107391758B (en) | Database switching method, device and equipment | |
| CN102708165B (en) | Document handling method in distributed file system and device | |
| CN103019960B (en) | Distributed caching method and system | |
| US8832113B2 (en) | Data management apparatus and system | |
| CN103095687B (en) | metadata processing method and device | |
| CN106649403B (en) | Index implementation method and system in file storage | |
| CN110334094B (en) | Data query method, system, device and equipment based on inverted index | |
| CN103186554A (en) | Distributed data mirroring method and data storage node | |
| CN109522283B (en) | Method and system for deleting repeated data | |
| CN104679665A (en) | Method and system for achieving block storage of distributed file system | |
| US20130031221A1 (en) | Distributed data storage system and method | |
| CN111858676A (en) | Data processing method and device | |
| CN104735110A (en) | Metadata management method and system | |
| CN102664914A (en) | IS/DFS-Image distributed file storage query system | |
| US20150169623A1 (en) | Distributed File System, File Access Method and Client Device | |
| CN110147203B (en) | File management method and device, electronic equipment and storage medium | |
| CN111930556A (en) | Backup method and device for distributed storage data | |
| CN107506466B (en) | Small file storage method and system | |
| CN104054076A (en) | Data storage method, database storage node failure processing method and apparatus | |
| CN101344882B (en) | Data query method, insertion method and deletion method | |
| CN104516945A (en) | Hadoop distributed file system metadata storage method based on relational data base |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| ASS | Succession or assignment of patent right |
Owner name: SHENGQU INFORMATION TECH (SHANGHAI) CO., LTD. Free format text: FORMER OWNER: SHANDA NETWORKING CO., LTD. Effective date: 20130918 |
|
| C41 | Transfer of patent application or patent right or utility model | ||
| COR | Change of bibliographic data |
Free format text: CORRECT: ADDRESS; FROM: 201203 PUDONG NEW AREA, SHANGHAI TO: 200241 MINHANG, SHANGHAI |
|
| TA01 | Transfer of patent application right |
Effective date of registration: 20130918 Address after: 200241 No. 1, building 690, blue wave road, Zhangjiang hi tech park, Shanghai Applicant after: Shengqu Information Technology (Shanghai) Co., Ltd. Address before: 201203 712-A room, No. 625 Zhangjiang Road, Shanghai, Pudong New Area Applicant before: Shanda computer (Shanghai) Co., Ltd. |
|
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20121226 |