Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberWO2016029744 A1
Publication typeApplication
Application numberPCT/CN2015/083226
Publication date3 Mar 2016
Filing date2 Jul 2015
Priority date27 Aug 2014
Also published asCN104202387A
Publication numberPCT/2015/83226, PCT/CN/15/083226, PCT/CN/15/83226, PCT/CN/2015/083226, PCT/CN/2015/83226, PCT/CN15/083226, PCT/CN15/83226, PCT/CN15083226, PCT/CN1583226, PCT/CN2015/083226, PCT/CN2015/83226, PCT/CN2015083226, PCT/CN201583226, WO 2016/029744 A1, WO 2016029744 A1, WO 2016029744A1, WO-A1-2016029744, WO2016/029744A1, WO2016029744 A1, WO2016029744A1
Inventors刘洋, 张克星
Applicant华为技术有限公司
Export CitationBiBTeX, EndNote, RefMan
External Links: Patentscope, Espacenet
Metadata recovery method and relevant device
WO 2016029744 A1
Abstract
Disclosed are a metadata recovery method and a relevant device. The method can comprise: receiving a write operation request for a target file, wherein the target file is saved in multiple data nodes; acquiring metadata of the target file from a namenode, the metadata of the target file containing information about the multiple data nodes; generating metadata recovery information according to the metadata of the target file; and sending the metadata recovery information to at least one data node of the multiple data nodes, the metadata recovery information being used for being saved by the at least one data node, and being sent to the namenode for recovering the metadata when abnormality occurs to the namenode. The embodiments of the present invention can recover metadata when abnormality occurs to an NN.
Images(13)
Previous page
Next page
Claims(24)  translated from Chinese
  1. 一种元数据恢复方法,其特征在于,包括: A meta data recovery method comprising:
    接收对目标文件的写操作请求,其中,所述目标文件保存在多个数据节点上; Receiving target file write request, wherein the target file data stored on multiple nodes;
    从元数据节点获取所述目标文件的元数据,所述目标文件的元数据包含所述多个数据节点的信息; Metadata node to obtain the file's metadata from the target, the target file's metadata contains information data of the plurality of nodes;
    根据所述目标文件的元数据生成元数据恢复信息; Metadata is information generated based on the metadata of the target file;
    向所述多个数据节点中的至少一个数据节点发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个数据节点进行保存,并在所述元数据节点出现异常后被发送到所述元数据节点进行所述元数据的恢复。 Sending the metadata information to recover at least one data node of said plurality of nodes in the data, the metadata information is used by the recovery of the at least one data node stored, with an exception after the metadata node transmitting the metadata node to recover the metadata.
  2. 如权利要求1所述的方法,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; The method as claimed in claim 1, characterized in that the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the data of the plurality of nodes;
    则,所述从元数据节点获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息包括: Then, the acquisition of the target file metadata from the metadata node, recovery information includes the target file metadata to generate metadata:
    从所述元数据节点获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的数据节点的信息; Get node of the plurality of data blocks of metadata from the metadata, the metadata for each data block contains information about each of the data blocks and each data block where the data nodes;
    根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The meta data for each of said data blocks, each data block generates the metadata recovery information.
  3. 如权利要求2所述的方法,其特征在于,所述从所述元数据节点获取所述多个数据块的元数据包括: The method as claimed in claim 2, wherein said acquiring a plurality of data blocks of metadata from the metadata node comprises:
    依次接收所述元数据节点发送的所述每个数据块的元数据;或者,接收所述元数据节点根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 Sequentially receiving the metadata node sends the metadata of each data block; or the metadata node receives the metadata request to be transmitted to all the data blocks according to the write target file.
  4. 如权利要求2或3所述的方法,其特征在于,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: 2 or 3 as claimed in claim wherein the received metadata comprises metadata metadata least the first data block and second data block, wherein:
    所述生成的所述第一数据块的元数据恢复信息包括: Said metadata generated data blocks to recover the first information includes:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;
    所述生成的所述第二数据块的元数据恢复信息包括: Generating said metadata of said data blocks to recover the second information comprises:
    所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.
  5. 一种元数据恢复方法,其特征在于,包括: A meta data recovery method comprising:
    接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从元数据节点获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; Metadata receiving client sends the target file recovery information, and storing the recovery information metadata, wherein the metadata information is restored to the client at the time of receiving the target file write request from the metadata node the metadata acquired target file, and recover the information based on metadata generated metadata;
    在所述元数据节点出现异常后,向所述元数据节点发送所述元数据恢复信息,元数据恢复信息被所述元数据节点用于进行所述元数据的恢复。 After the metadata node is abnormal, sending the information to restore the metadata node metadata, metadata information is restored recover the metadata node for performing the metadata.
  6. 如权利要求5所述的方法,其特征在于,所述目标文件被划分为多个数据块,所述接收客户端发送的目标文件的元数据恢复信息,包括: The method as claimed in claim 5, characterized in that the target file is divided into a plurality of data blocks, the client sends the received metadata object file recovery information, comprising:
    接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Receiving the data block sent by the client to the target file metadata recovery information, the metadata of the data block recovery information of the client to recover information from the metadata of the data block to generate metadata.
  7. 如权利要求6所述的方法,其特征在于,所述接收所述客户端发送的所述目标文件的数据块的元数据恢复信息包括: The method according to claim 6, characterized in that the meta-data of data blocks of the target files sent by the client receiving the recovery information comprising:
    依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata sequentially receiving a plurality of data blocks sent by the client end of the object file recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information sent at once.
  8. 如权利要求6或7所述的方法,其特征在于,所述接收的元数据恢复信息至少包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: The method as claimed in 6 or claim 7, wherein said recovery information received metadata comprises at least a first data block of metadata and metadata information recovery data blocks to recover a second information, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;
    所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:
    所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.
  9. 一种元数据恢复方法,其特征在于,包括: A meta data recovery method comprising:
    接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个数据节点上; Receiving client sent to the target file write request, wherein the target file data stored on multiple nodes;
    生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个数据节点的信息; Generating the target file's metadata, the client sends to the metadata, wherein the metadata contains information on the plurality of data nodes;
    当本端出现异常时,接收所述多个数据节点中的至少一个数据节点发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个数据节点进行保存的元数据恢复信息。 When the end of the abnormal, metadata receiving the plurality of data nodes in at least one node sends data recovery information, and recover the metadata information retrieval based on the metadata, wherein the metadata information is restored the client-generated based on the metadata, sent by the client to the at least one data node metadata saved recovery information.
  10. 如权利要求9所述的方法,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; The method as claimed in claim 9, characterized in that the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the data of the plurality of nodes;
    则所述生成所述目标文件的元数据,并向所述客户端发送所述元数据包括: The generator is the target data file, the client sends to the metadata comprises:
    生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的数据节点的信息; Generating the plurality of data blocks of metadata to the client sends the metadata of the plurality of data blocks, the metadata for each data block contains information about each of the data blocks and each data block is located information data nodes;
    所述接收所述多个数据节点中的至少一个数据节点发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,包括: Receiving said data of said plurality of nodes the metadata node sends at least one data recovery information, and restore the recovery information metadata according to the metadata, comprising:
    接收所述多个数据节点中的至少一个数据节点发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Metadata receiving said plurality of data blocks of said plurality of data nodes transmit at least one data node in the recovery information and the recovery information recovery target file based on metadata of said plurality of data blocks of metadata.
  11. 如权利要求10所述的方法,其特征在于,所述向所述客户端发送 所述多个数据块的元数据包括: The method as claimed in claim 10, wherein said transmitting said client metadata to the plurality of data blocks comprising:
    依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially sends to the client the metadata for the plurality of data blocks; or request a one-time write operation according to the target file metadata for all data blocks to send to the client.
  12. 如权利要求10或11所述的方法,其特征在于,所述多个数据块至少包括第一数据块和第二数据块,其中: The method according to 10 or claim 11, wherein said plurality of data blocks include at least a first data block and second data block, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;
    所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:
    所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.
  13. 一种客户端,其特征在于,包括:接收单元、获取单元、生成单元和发送单元,其中: A client-side, characterized by comprising: a receiving unit, acquisition unit, a generating unit and a sending unit, wherein:
    所述接收单元,用于接收对目标文件的写操作请求,其中,所述目标文件保存在多个数据节点上; The receiving unit for receiving the target file write request, wherein the target file data stored on multiple nodes;
    所述获取单元,用于从元数据节点获取所述目标文件的元数据,所述目标文件的元数据包含所述多个数据节点的信息; The acquisition unit for acquiring the target node from the metadata file metadata, the target file's metadata contains information about the plurality of data nodes;
    所述生成单元,用于根据所述目标文件的元数据生成元数据恢复信息; Said generating means for generating metadata based on metadata of said Recovery Information of the target files;
    所述发送单元,用于向所述多个数据节点中的至少一个数据节点发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个数据节点进行保存,并在所述元数据节点出现异常后被发送到所述元数据节点进行所述元数据的恢复。 The transmitting means for transmitting the meta data information to recover at least one data node of said plurality of nodes in the data, the metadata information is used by the recovery of the stored at least one data node, and the the metadata node is abnormal after the metadata node sends to recover the metadata.
  14. 如权利要求13所述的客户端,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; 13 The client as claimed in claim, characterized in that the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the data of the plurality of nodes;
    则,所述获取单元用于从所述元数据节点获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的数据节点的信息; Then, the acquisition unit for acquiring said plurality of data blocks of metadata from the metadata node, the metadata for each data block contains information about each of the data blocks and each data block where the data node information;
    所述生成单元用于根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 Said means for generating the meta data for each of said data blocks, each data block generates the metadata recovery information.
  15. 如权利要求14所述的客户端,其特征在于,所述获取单元用于依次接收所述元数据节点发送的所述每个数据块的元数据;或者,所述获取单元用于接收所述元数据节点根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 14 The client as claimed in claim wherein said means for obtaining the metadata of the metadata sequentially receiving node to send each data block; or the acquisition unit for receiving the metadata node requests at once metadata for all data block to be transmitted according to the write target file.
  16. 如权利要求14或15所述的客户端,其特征在于,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: 14 or 15 according to the client as claimed in claim wherein the received metadata comprises metadata metadata least the first data block and second data block, wherein:
    所述生成的所述第一数据块的元数据恢复信息包括: Said metadata generated data blocks to recover the first information includes:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;
    所述生成的所述第二数据块的元数据恢复信息包括: Generating said metadata of said data blocks to recover the second information comprises:
    所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.
  17. 一种数据节点,其特征在于,包括:接收单元和发送单元,其中: A data node, characterized by comprising: a receiving unit and a sending unit, wherein:
    所述接收单元,用于接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从元数据节点获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; The receiving unit for receiving data sent by the client metadata object file recovery information, and storing the recovery information metadata, wherein the metadata information is restored in the client receives the write target file when requested, the metadata of the target file's metadata acquired from the node, and recover information based on the metadata generated by the metadata;
    所述发送单元,用于在所述元数据节点出现异常后,向所述元数据节点发送所述元数据恢复信息,元数据恢复信息被所述元数据节点用于进行所述元数据的恢复。 The transmitting means for returning said metadata node is abnormal, sending the information to restore the metadata node metadata, metadata information is restored recover the metadata node for performing the metadata .
  18. 如权利要求17所述的数据节点,其特征在于,所述目标文件被划分为多个数据块,所述接收单元用于接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 A metadata node as claimed in claim 17, characterized in that the target file is divided into a plurality of blocks of data, said receiving means for receiving the data block sent by the client of the recovery target files information, meta data of the data blocks to recover the information to the client based on the metadata information to restore the data block of the metadata.
  19. 如权利要求18所述的数据节点,其特征在于,所述接收单元用于 依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,所述接收单元用于接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Data node according to claim 18, wherein said plurality of data blocks received metadata unit for sequentially receiving the client endpoint of the object file recovery information; Alternatively, the receiving unit with metadata for receiving a plurality of data blocks sent by the client of the disposable object file recovery information.
  20. 如权利要求17或18所述的数据节点,其特征在于,所述接收的元数据恢复信息至少包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: 17 or 18, the data node according to claim, wherein said recovery information received metadata comprises at least a first data block of metadata and metadata information recovery data blocks to recover a second information, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;
    所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:
    所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.
  21. 一种元数据节点,其特征在于,包括:接收单元、生成单元和恢复单元,其中: A meta data node, characterized by comprising: a receiving unit, a generating unit and a recovery unit, wherein:
    所述接收单元,用于接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个数据节点上; The receiving unit for receiving client sent to the target file write request, wherein the target file data stored on multiple nodes;
    所述生成单元,用于生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个数据节点的信息; Said generating means, for generating the target file's metadata, the client sends to the metadata, wherein the metadata contains information on the plurality of data nodes;
    所述恢复单元,用于当本端出现异常时,接收所述多个数据节点中的至少一个数据节点发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个数据节点进行保存的元数据恢复信息。 The recovery unit for, when the end of the abnormal, metadata receiving the plurality of data nodes in at least one node sends data recovery information, and recover the metadata information retrieval based on the metadata, wherein, the meta data recovery information is generated by the client according to the metadata sent by the client to the at least one data node metadata saved recovery information.
  22. 如权利要求21所述的元数据节点,其特征在于,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个数据节点上; Metadata node as claimed in claim 21, characterized in that the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the data of the plurality of nodes;
    所述生成单元用于生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息 和所述每个数据块所在的数据节点的信息; The metadata generation unit for generating a plurality of data blocks to the client sends the metadata of the plurality of data blocks, each block of metadata containing information for each data block and the said information for each data block where the data nodes;
    所述恢复单元用于当本端出现异常时,接收所述多个数据节点中的至少一个数据节点发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 The recovery unit is used when the local abnormal, metadata receiving the plurality of data blocks of the plurality of data nodes in at least one node sends data recovery information and the meta data of said plurality of blocks data recovery information recovery of the target file's metadata.
  23. 如权利要求21所述的元数据节点,其特征在于,所述生成单元用于生成所述多个数据块的元数据,并依次向所述客户端发送所述多个数据块的元数据;或者,所述生成单元用于生成所述多个数据块的元数据,并根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Metadata node as claimed in claim 21, wherein said data generator means for generating said plurality of data blocks, and sequentially sends the plurality of data blocks of the metadata to the client; Alternatively, the generating means for generating a plurality of data blocks of the metadata and sends the metadata at once all data blocks to the client in accordance with the write request to the target file.
  24. 如权利要求10或11所述的元数据节点,其特征在于,所述多个数据块至少包括第一数据块和第二数据块,其中: 10 or 11, wherein the metadata node as claimed in claim, wherein said plurality of data blocks include at least a first data block and second data block, wherein:
    所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:
    所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;
    所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:
    所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.
Description  translated from Chinese
一种元数据恢复方法及相关装置 A meta data recovery method and related apparatus 技术领域 TECHNICAL FIELD

本发明涉及通信领域,尤其涉及一种元数据恢复方法及相关装置。 The present invention relates to the field of communications, particularly to a method and related meta data recovery apparatus.

背景技术 Background technique

Hadoop是目前通信领域中应用十分广泛的分布式系统架构,在Hadoop中包含一个非常重要的存储组件,即Hadoop分布式文件系统(Hadoop Distributed File System,简称HDFS)。 Hadoop is currently in the field of communications is widely distributed system architecture, contains a very important component stored in Hadoop, that Hadoop Distributed File System (Hadoop Distributed File System, referred to HDFS). 其中,HDFS包括元数据节点(NameNode,简称NN)和数据节点(DataNode,简称DN),NN主要用来提供文件的元数据服务,该服务中NN在内存会生成该文件的元数据,元数据可以表示该文件的数据结构,而DN主要用来存储文件的实际数据。 Wherein, HDFS including metadata node (NameNode, referred to as NN) and a data node (DataNode, abbreviated DN), NN mainly used to provide file metadata service that the NN generates the file in the memory will be metadata metadata It may represent a data structure of the file, and the DN is mainly used to store the actual data files. 在实际应用中当NN出现异常时,这样该NN就会出现元数据丢失的问题,以导致该NN失效。 In practice, when the NN abnormal, so that the NN will be metadata loss problem to cause the failure of NN.

其中,目前主要通过如下技术解决上述问题: Wherein, the key to solve the above problems by the following techniques:

在NN内部开启一个定时线程,该线程定期将NN内存中的元数据写到本地磁盘或者远端存储设备形成一个特定文件。 NN open in a timed internal thread that periodically NN-memory metadata written to the local disk or a remote storage device forms a specific file. 当NN出现异常后,NN在重新启动过程中加载该特定文件,以恢复之前的某一时段的元数据,以实现NN恢复到这一时段的内存状态。 When abnormal NN, NN load the specific file in the restart process, meta data for a period of time to recover before to achieve NN restore to this period memory status.

但上述技术方案中,由于是定期的存储NN内的元数据,例如:每隔十分钟存储一次元数据,这样当NN出现异常的时间点还未到十分钟一次的存储时间点时,那么NN内的出现异常时的元数据不会存储,从而这部分元数据就无法恢复。 However, the above aspect, since the metadata stored NN regularly within, for example: once every ten minutes to store metadata, so that when the time has not yet NN abnormal storage time to ten minutes once, then the NN metadata is not stored within the exception occurred, so this part of the meta-data can not be recovered. 可见,上述技术方案中无法恢复NN异常时的元数据。 Visible, it can not recover the metadata NN exception of the above-mentioned technical solution.

发明内容 SUMMARY

本发明提供了一种元数据恢复方法及相关装置,可以恢复NN异常时的元数据。 The present invention provides a method and associated meta data recovery device that can recover the metadata NN when an abnormality.

第一方面,本发明提供一种元数据恢复方法,包括: A first aspect, the present invention provides a meta data recovery method, comprising:

接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上; Receiving target file write request, wherein the target file is saved on multiple DN;

从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息; Get the target document from NN metadata, the target file metadata contains the DN of the plurality of information;

根据所述目标文件的元数据生成元数据恢复信息; Metadata is information generated based on the metadata of the target file;

向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 To the at least one of said plurality of DN DN transmitting the metadata to restore information, restore the metadata information for the at least one DN is saved and the NN abnormal then sent to the NN to recover the metadata.

在第一方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In a first aspect of a first possible implementation, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,所述从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息包括: Then, the acquisition of the target file metadata from NN, recovery information including the meta data of the target file to generate the metadata:

从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Get the plurality of metadata blocks from the NN, the metadata for each data block contains information DN information for each data block and said each data block is located;

根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The meta data for each of said data blocks, each data block generates the metadata recovery information.

结合第一方面的第一种可能的实现方式,在第二种可能实现方式中,所述从所述NN获取所述多个数据块的元数据包括: Combining a first possible implementation of the first aspect, in a second possible embodiment, said acquiring said plurality of data blocks from the NN metadata comprises:

依次接收所述NN发送的所述每个数据块的元数据;或者,接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 NN transmitting said sequentially receives the metadata of each data block; or NN receiving the request metadata for all data blocks in one transmission operation according to the write target file.

结合第一方面的第一种可能的实现方式或者第二种可能的实现方式,在第三种可能实现方式中,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: Combining a first possible implementation of the first aspect or the second possible implementation, in a third possible implementation, the received metadata comprises metadata for at least a first data block and second data block meta-data, including:

所述生成的所述第一数据块的元数据恢复信息包括: Said metadata generated data blocks to recover the first information includes:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述生成的所述第二数据块的元数据恢复信息包括: Generating said metadata of said data blocks to recover the second information comprises:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

第三方面,本发明提供一种元数据恢复方法,包括: A third aspect, the present invention provides a meta data recovery method, comprising:

接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; Receiving target file sent by the client metadata recovery information, and storing the recovery information metadata, wherein the metadata information is restored to the client at the time of receiving the target file write request, obtained from NN the target file's metadata, and recover the information based on metadata generated by the metadata.;

在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 After the NN abnormal, sending the information to the meta data recovery NN, metadata restore the recovery information is used for the NN metadata.

在第二方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述接收客户端发送的目标文件的元数据恢复信息,包括: The first possible implementation of the second aspect, the target file is divided into a plurality of data blocks, the client sends the received metadata object file recovery, including:

接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Receiving the data block sent by the client to the target file metadata recovery information, the metadata of the data block recovery information of the client to recover information from the metadata of the data block to generate metadata.

结合第二方面的第一种可能的实现方式,在第二方面的第二种可能的实现方式中,所述接收所述客户端发送的所述目标文件的数据块的元数据恢复信息包括: Combining a first possible implementation of the second aspect, in the second aspect of a second possible implementation, the received data block is the metadata of the target client sends the file recovery information comprising:

依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata sequentially receiving a plurality of data blocks sent by the client end of the object file recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information sent at once.

结合第二方面的第一种可能的实现方式或者第二种可能的实现方式中,在第二方面的第三种可能的实现方式中,所述接收的元数据恢复信息至少包 括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Combining a first possible implementation of the second aspect or the second possible implementation, in the second aspect of the third possible implementation, the recovery of the received metadata information includes at least a first data block the metadata information and the second recovery data recovery block metadata information, wherein:

所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

第三方面,本发明提供一种元数据恢复方法,包括: A third aspect, the present invention provides a meta data recovery method, comprising:

接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上; Receiving client sent to the target file write request, wherein the target file is saved on multiple DN;

生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息; Generating the target file's metadata, the client sends to the metadata, wherein the metadata contains the DN of the plurality of information;

当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 When the end of the abnormal, metadata receiving the plurality of DN DN send at least one message of recovery, restoration and recovery information of the metadata according to the metadata, wherein the metadata is the recovery information client-generated based on the metadata, sent by the client to the at least one DN to save metadata recovery information.

在第三方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In a third aspect to a first possible implementation, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则所述生成所述目标文件的元数据,并向所述客户端发送所述元数据包括: The generator is the target data file, the client sends to the metadata comprises:

生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Generating the plurality of data blocks of metadata to the client sends the metadata of the plurality of data blocks, the metadata for each data block contains information about each of the data blocks and each data block is located the DN information;

所述接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,包括: Receiving said plurality of said at least one DN DN metadata transmitted recovery information, and restore the recovery information metadata according to the metadata, comprising:

接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Receiving said plurality of data blocks of said at least one of the plurality of DN DN metadata transmitted recovery information and the recovery information recovery target file based on metadata of said plurality of data blocks of metadata.

结合第三方面的第一种可能的实现方式,在第三方面的第二种可能的实现方式中,所述向所述客户端发送所述多个数据块的元数据包括: Combining a first possible implementation of the third aspect, in the third aspect of a second possible implementation, the client sends the plurality of data blocks of said metadata includes:

依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially sends to the client the metadata for the plurality of data blocks; or request a one-time write operation according to the target file metadata for all data blocks to send to the client.

结合第三方面的第一种可能的实现方式或者第二种可能的实现方式,在第三方面的第三种可能的实现方式中,所述多个数据块至少包括第一数据块和第二数据块,其中: Combined with a first possible implementation of the second or third aspect of the possible implementation of the third aspect of the third possible implementation, the plurality of data blocks including at least a first and a second data block data block, wherein:

所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

第四方面,本发明提供一种客户端,包括:接收单元、获取单元、生成单元和发送单元,其中: A fourth aspect, the present invention provides a client, comprising: a receiving unit, acquisition unit, a generating unit and a sending unit, wherein:

所述接收单元,用于接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上; The receiving unit for receiving the target file write request, wherein the target file is saved on multiple DN;

所述获取单元,用于从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息; The acquisition unit for acquiring the target data file from the meta NN, the destination file metadata contains the DN of the plurality of information;

所述生成单元,用于根据所述目标文件的元数据生成元数据恢复信息; Said generating means for generating metadata based on metadata of said Recovery Information of the target files;

所述发送单元,用于向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所 述NN出现异常后被发送到所述NN进行所述元数据的恢复。 The sending unit for supplying the at least one of said plurality of DN DN transmitting the metadata to restore information, restore the metadata information for the at least one DN is saved, and abnormal in the NN NN then sent to the recovery of the metadata.

在第四方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In a fourth aspect of one possible implementation of the first embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,所述获取单元用于从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Then, the acquisition unit for acquiring metadata of said plurality of data blocks from the NN, the metadata of each data block contains the DN information and the information for each of said data blocks of each data block is located;

所述生成单元用于根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 Said means for generating the meta data for each of said data blocks, each data block generates the metadata recovery information.

结合第四方面的第一种可能的实现方式,在第四方面的第二种可能的实现方式中,所述获取单元用于依次接收所述NN发送的所述每个数据块的元数据;或者,所述获取单元用于接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 Combining a first possible implementation of the fourth aspect, in the fourth aspect of a second possible implementation, the metadata obtaining means for sequentially receiving said transmitted the NN for each data block; Alternatively, the acquisition unit for receiving the NN in accordance with the target file write request to be sent to all of the metadata data blocks.

结合第四方面的第一种可能的实现方式或者第二种可能的实现方式,在第四方面的第三种可能的实现方式中,所述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,其中: Combining a first possible implementation of the fourth aspect or the second possible implementation, in the fourth aspect of the third possible implementation, the received metadata comprises metadata for at least a first data block and meta data of the second data block, wherein:

所述生成的所述第一数据块的元数据恢复信息包括: Said metadata generated data blocks to recover the first information includes:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述生成的所述第二数据块的元数据恢复信息包括: Generating said metadata of said data blocks to recover the second information comprises:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

第五方面,本发明提供一种DN,包括:接收单元和发送单元,其中: Fifth aspect, the present invention provides a the DN, comprising: a receiving unit and a sending unit, wherein:

所述接收单元,用于接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据 所述元数据生成的元数据恢复信息.; The receiving unit for receiving data sent by the client metadata object file recovery information, and storing the recovery information metadata, wherein the metadata information is restored in the client receives the write target file when requested, NN metadata acquired from the target file, and recover information according to the metadata generated by the metadata.;

所述发送单元,用于在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 The transmission unit, after the NN for abnormal, sending the information to the meta data recovery NN, metadata restore the recovery information is used for the NN metadata.

在第五方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述接收单元用于接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 In a fifth aspect of one possible implementation of the first embodiment, the target file is divided into a plurality of blocks of data, said receiving means for receiving the data block sent by the client to the target file metadata recovery information, meta data of the data blocks to recover the information to the client based on the metadata information to restore the data block of the metadata.

结合第五方面的第一种可能的实现方式,在第五方面的第二种可能的实现方式中,所述接收单元用于依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,所述接收单元用于接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Combining a first possible implementation of the fifth aspect, in the fifth aspect of the second possible implementation, the means for receiving a plurality of data blocks sequentially receiving the client endpoint of the target file recovery metadata information; Alternatively, the meta-data receiving means for receiving said plurality of data blocks sent by the client disposable object file recovery information.

结合第五方面的第一种可能的实现方式或者第二种可能的实现方式,在第三种可能的实现方式中,所述接收的元数据恢复信息至少包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Combining a first possible implementation of the fifth aspect or the second possible implementation, in a third possible implementation, the recovery of the received metadata information comprises at least a first data block of the metadata recovery information and meta data of the second block of data recovery information, including:

所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

第六方面,本发明提供一种NN,包括:接收单元、生成单元和恢复单元,其中: A sixth aspect, the present invention provides a NN, comprising: a receiving unit, a generating unit and a recovery unit, wherein:

所述接收单元,用于接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上; The receiving unit for receiving client sent to the target file write request, wherein the target file is saved on multiple DN;

所述生成单元,用于生成所述目标文件的元数据,并向所述客户端发送 所述元数据,其中,所述元数据包含所述多个DN的信息; Said generating means, for generating the target file's metadata, the client sends to the metadata, wherein the metadata contains the DN of the plurality of information;

所述恢复单元,用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 The recovery unit for, when the end of the abnormal, metadata receiving the plurality of DN DN send at least one message of recovery, restoration and recovery information of the metadata according to the metadata, wherein metadata is the information generated by the clients according to the metadata sent by the client to the at least one DN to save metadata recovery information.

在第六方面的第一种可能的实现方式中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In a sixth aspect to a first possible implementation, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

所述生成单元用于生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; The metadata generation unit for generating a plurality of data blocks to the client sends the metadata of the plurality of data blocks, each block of metadata containing information for each data block and the said DN information for each data block is located;

所述恢复单元用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 The recovery unit is used when the local abnormal, metadata of the plurality of data blocks of said plurality of receiving at least one DN DN send recovery information, and recover from the metadata of the plurality of data blocks information recovery of the target file's metadata.

结合第六方面的第一种可能的实现方式,在第二种可能的实现方式中,所述生成单元用于生成所述多个数据块的元数据,并依次向所述客户端发送所述多个数据块的元数据;或者,所述生成单元用于生成所述多个数据块的元数据,并根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Combined with a first possible implementation of the sixth aspect, in a second possible implementation, the generating unit for generating a plurality of data blocks of metadata, and in turn send the to the client metadata plurality of data blocks; or the generation means for generating a plurality of data blocks of metadata and data blocks to send it all the client requests a write operation according to the target file metadata .

结合第六方面的第一种可能的实现方式或者第二种可能的实现方式,在第三种可能的实现方式中,所述多个数据块至少包括第一数据块和第二数据块,其中: Combined with a first possible implementation of the sixth aspect or the second possible implementation, the third possible implementation, the plurality of data blocks including at least a first data block and a second block, wherein :

所述第一数据块的元数据恢复信息包括: The metadata data blocks to recover the first information includes:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息包括: The metadata data blocks to recover the second information comprises:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

上述技术方案中,接收对目标文件的写操作请求,再从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息,再向所述多个DN中的至少一个DN发送所述元数据恢复信息。 The technical proposal, the reception of the target file write request, then get metadata from the target file NN, generate metadata based on metadata information to restore the target file, again at least in the more DN a DN transmit the meta data recovery information. 这样该元数据恢复信息就可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 So that the recovery of metadata information can be used to save at least one DN and the NN abnormal then sent to the NN to recover the metadata. 从而可以恢复NN出异常时的元数据。 NN which can restore the metadata when an abnormality.

附图说明 BRIEF DESCRIPTION

为了更清楚地说明本发明实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。 In order to more clearly illustrate the embodiments or the prior art technical solutions embodiment of the present invention, it will implement the following figures for the cases described in the prior art or the need to use a simple introduction. Apparently, the following description of the drawings are only the present invention is to some embodiments, those of ordinary skill in speaking, without creative efforts of the premise, but also can derive other drawings from these drawings.

图1是本发明实施例提供的一种可选的元数据的示意图; Figure 1 is a schematic illustration of an alternative embodiment of metadata provided by the present invention;

图2是本发明实施例提供的一种元数据恢复方法的流程示意图; Figure 2 is a schematic flow diagram of embodiments of the present invention to provide a meta-data examples of recovery methods;

图3是本发明实施例提供的另一种元数据恢复方法的流程示意图; Figure 3 is a schematic flow diagram of another embodiment of the present invention is to provide patients with metadata recovery method;

图4是本发明实施例提供的一种可选的数据信息的示意图; 4 is a schematic embodiment of the invention provides an alternative data information;

图5是本发明实施例提供的另一种元数据恢复方法的流程示意图; Figure 5 is another example of metadata restore method available in a schematic flow diagram of embodiment of the present invention;

图6是本发明实施例提供的另一种元数据恢复方法的流程示意图; Figure 6 is a schematic flow diagram of another embodiment of the present invention is to provide patients with metadata recovery method;

图7是本发明实施例提供的另一种元数据恢复方法的流程示意图; Figure 7 is a schematic flow diagram of another embodiment of the present invention is to provide patients with metadata recovery method;

图8是本发明实施例提供的一种客户端的结构示意图; Figure 8 is a block schematic of one embodiment of the present invention to provide customers of the end;

图9是本发明实施例提供的一种DN的结构示意图; FIG. 9 is a block diagram of an embodiment of the invention the DN provided;

图10是本发明实施例提供的一种NN的结构示意图; FIG. 10 is a block diagram of an NN, according to an embodiment of the present invention;

图11是本发明实施例提供的另一种元数据恢复设备的结构示意图; FIG. 11 is a block diagram of another example of metadata provided by the recovery device embodiment of the present invention;

图12是本发明实施例提供的一种DN设备的结构示意图; FIG 12 is a schematic structural view of embodiment of the present invention, one kind of device provided DN;

图13是本发明实施例提供的一种NN设备的结构示意图。 FIG. 13 is a block diagram of the present embodiment invented a device provided by the NN.

具体实施方式 detailed description

下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本发明一部分实施例,而不是全部的实施例。 The present invention will now be combined with the implementation of the accompanying drawings, were clear examples of technical solutions of the present invention, a complete description, it is clear that the described embodiments are merely part of the embodiments of the present invention, but not all embodiments. 基于本发明中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本发明保护的范围。 Based on the embodiments of the present invention, those of ordinary skill in not making all other embodiments without creative efforts obtained, are within the scope of the present invention is protected.

本发明实施例中,元数据可以是用于表示文件数据结构的数据,其中,文件的元数据具体可以表示该文件所在的HDFS系统的版本信息和该文件的属性信息,例如:文件的元数据包括的HDFS系统的版本信息可以包括如下一项或者多项: Embodiment of the invention, the metadata can be used to represent data file data structure, wherein the metadata file attribute information may represent a specific version of the file system and the HDFS file, for example: the file's metadata HDFS version information included in the system may include one or more of:

HDFS系统的版本信息(例如:imgVersion)、名称空间标识(例如:namespaceID)、HDFS系统的文件个数(例如:numFiles)和HDFS系统的时间戳(例如:genStamp); HDFS system version information (for example: imgVersion), namespace identifier (for example: namespaceID), the number of HDFS file system (for example: numFiles) and HDFS system timestamp (for example: genStamp);

文件的元数据包括的该文件的属性信息可以包括如下一项或者多项: Attribute information of the file metadata file includes may include one or more of:

文件的路径(例如:Path)、文件的副本数(例如:replicas)、文件的访问时间(例如:atime)、文件的每个数据块大小(例如:blockSize)、文件的数据块的块数(例如:numBlock)、文件占据的空间的限制信息(例如:nsQuota和dsQuota)和用户权限信息(例如:username、group和perm)。 Path of the file (for example: Path), the number of copies of the file (for example: replicas), file access time (for example: atime), each data block size of the file (for example: blockSize), the number of blocks of data blocks of the file ( for example: numBlock), limiting the space occupied by the file information (for example: nsQuota and dsQuota) and user privileges information (for example: username, group and perm).

另外,当文件包括多个数据块时,文件的元数据还可以包括各数据块的元数据,其中,数据块的元数据包括该数据块所属的文件的属性信息和该数据块的属性信息;其中,数据块的属性信息可以包括如下一项或者多项: Further, when the document includes a plurality of data blocks, the metadata file can also include metadata for each data block, wherein the metadata includes attribute information block attribute information of the data block belongs file and the data block; wherein the attribute information data blocks may include one or more of:

数据块标识(例如:BlockId)、数据块存储的数据量(例如:numBytes)和数据块的时间戳(例如:genStamp)。 Identification data block (for example: BlockId), the amount of data (for example: numBytes) data blocks and data blocks stored timestamp (for example: genStamp). 另外,文件的属性信息上面已经介绍,此处不重复说明。 Further, the attribute information of the file has been described above, description will not be repeated here.

例如:文件的元数据可以如图1所示,文件的元数据可以包括图1中的 三行数据,其中,第一行表示HDFS系统的版本信息,第二行表示文件的属性信息,第三行表示数据块的元数据。 For example: Metadata files can be shown in Figure 1, the metadata file can include three lines of data in Figure 1, wherein the first line indicates the version information HDFS system, the second row represents attribute information file, and the third row represents the metadata block. 即文件的元数据可以包括HDFS系统的版本信息、文件的属性信息和数据块的元数据。 That metadata file may include version information for HDFS systems, metadata attribute information and data blocks of the file.

另外,本发明实施例应用的场景可以是单个设备,即该设备上包括HDFS的客户端、至少一个NN和至少一个DN;或者本发明实施例应用的场景可以是设备集群,即HDFS的客户端、至少一个NN和至少一个DN部署在多个设备上。 In addition, the scene embodiment of the invention application may be a single device, which includes HDFS client on the device, at least one NN and at least one DN; or scenes embodiment of the invention application may be a device cluster that HDFS client at least one of NN and at least one DN deployed on multiple devices. 其中,上述设备可以是网络侧设备,例如:基站、服务器、控制器等设备,上述设备还可以是用户侧设备,例如:计算机、平板电脑、手机、电子阅读器、遥控器、笔记本电脑、车载设备、网络电视、可穿戴设备等具有网络功能的智能设备。 Wherein the device may be a network-side device, such as: base stations, servers, controllers and other equipment, these devices can also be a client-side equipment, such as: computer, tablet, phone, e-reader, remote control, notebook computers, automotive equipment, network television, wearable devices and other network-enabled smart devices.

请参阅图2,图2是本发明实施例提供的一种元数据恢复方法的流程示意图,如图2所示,包括以下步骤: See Figure 2, Figure 2 is a metadata embodiment of the invention provides a method of restoring a schematic flow diagram, shown in Figure 2, comprising the steps of:

201、接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上。 201, receiving the target file write request, wherein the target file is saved on multiple DN.

上述目标文件保存在多个DN上可以是响应上述写操作请求,将上述目标文件保存在多个DN上,其中,该保存过程的时间不作限定,例如:可以是在步骤204之后,或者在步骤204之前。 Above the target file is stored on multiple DN may be in response to the write request, the above target file is saved on multiple DN, wherein the time of the save process is not as defined, for example: either after step 204, or step 204 before.

202、从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息。 202, get metadata from the target file NN, the metadata of the object file containing the plurality of DN information.

可选的,当接收到上述写操作请求时,本端就可以向NN申请用于保存目标文件的多个DN,从而NN就可以为该目标文件分配NN,以及生成该目标文件的元数据,并将该目标文件的元数据返回到上述本端。 Alternatively, when receiving the write request, to this end it can be used to save the target file NN apply multiple DN, so that can be allocated for the NN NN object files, and generate the metadata of the target file, and returns the metadata for that file to the target of the present terminal. 另外,上述多个DN的信息可以是各DN保存上述目标文件的路径信息,例如:上述介绍的文件的属性信息中的文件的路径。 Further, the plurality of DN information may be stored above the target file path information of each DN, such as: the introduction of the path attribute information file in the file.

203、根据所述目标文件的元数据生成元数据恢复信息。 203, recovery information is generated metadata from the metadata of the target file.

可选的,上述元数据恢复信息可以包括该目标文件的元数据,例如:包括目标文件的属性信息。 Alternatively, said recovery metadata information may include metadata that object file, such as: the attribute information comprises a destination file. 其中,该属性信息可以包括该目标文件的如下一项或者多项: Wherein, the attribute information may include one or more of the target file:

HDFS系统的版本信息、文件名、文件占据的空间的限制信息、用户权限信息、文件的每个数据块大小、文件的数据块的块数。 The number of data blocks in version HDFS system, each data block size of the file names, limit the space occupied by the file information, user rights information, file, file.

其中,上述文件名还可以携带有文件的路径信息。 Wherein the file name can also carry the file path information.

204、向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 204, to the at least one of the plurality of DN DN transmitting the metadata to restore information, restore the metadata information for the at least one DN is saved and the NN abnormal then sent to the said NN recover the metadata.

可选的,上述方法可以应用于HDFS的客户端(Client),即该客户端可以实现上述方法。 Alternatively, the above method may be applied to HDFS client (Client), that is, the client can achieve the above methods.

上述技术方案中,接收对目标文件的写操作请求,再从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息,再向所述多个DN中的至少一个DN发送所述元数据恢复信息。 The technical proposal, the reception of the target file write request, then get metadata from the target file NN, generate metadata based on metadata information to restore the target file, again at least in the more DN a DN transmit the meta data recovery information. 这样该元数据恢复信息就可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 So that the recovery of metadata information can be used to save at least one DN and the NN abnormal then sent to the NN to recover the metadata. 从而可以恢复NN出异常时的元数据。 NN which can restore the metadata when an abnormality.

请参阅图3,图3是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图3所示,包括以下步骤: Refer to FIG. 3, FIG. 3 is a flowchart of another embodiment of the present invention is provided in Example metadata schematic recovery method, shown in Figure 3, comprising the steps of:

301、接收对目标文件的写操作请求,其中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上。 301, receiving a request to write the target file, wherein the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of DN.

可选的,对目标文件进行数据写操作可以是以数据块为单位进行写操作,即一个目标文件可以被划分为多个数据块中,其中,每个数据块的大小 可以是相同。 Alternatively, the target file is a data write operation in data block write operation, i.e. a target file may be divided into a plurality of data blocks, wherein the size of each data block may be the same. 例如:每个数据块可以是64兆(MB)或者128MB等。 For example: each data block can be 64 megabytes (MB), etc., or 128MB. 另外,上述数据块的大小可以是数据块的存储空间大小,但在实际应用中一个文件的各数据块存储的该文件的数据量可以是不同的,例如:一个文件包含150MB数据,而每个数据块的大小为64MB,那么该文件的第一个数据块就可以存储该文件的64MB数据,而第二个数据块就可以存储该文件的64MB数据,而第三个数据块就可以存储该文件的22MB数据。 Further, the size of the data block may be a block of data storage space, but in practice the amount of data of each data block of a file stored in the file may be different, for example: a 150MB file contains data, and each data block size is 64MB, then the first block of the file can store 64MB of data of the file, while the second block of data can be stored in the 64MB data file, and the third block of data that can be stored 22MB data file.

302、从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 302, acquiring the plurality of data blocks of metadata from the NN, the metadata for each data block contains information DN information for each data block and said each data block is located.

可选的,当接收到上述写操作请求时,本端就可以向NN申请用于保存目标文件的多个数据块,以及保存数据块的多个DN。 Alternatively, when receiving the write request, to this end it can be used to save the target file NN apply a plurality of data blocks, and a plurality of data blocks stored DN. 这样从而NN就可以为该目标文件分配数据块,以及数据块所在的DN,以及生成每个数据块的元数据,并将每个数据块的元数据返回到上述本端。 DN thus NN so you can allocate data blocks for the target file, and data blocks are located, as well as generate metadata for each data block and return the metadata for each data block to the end of the present. 另外,上述数据块的信息可以是数据块的属性信息,例如:数据块标识和数据块的时间戳。 In addition, the information of the data blocks may be the attribute information data block, for example: timestamp data block ID and data block. 上述多个DN的信息可以是各DN保存数据块的路径信息,例如:上述介绍的文件的属性信息中的文件的路径。 DN of the plurality of information can be stored for each block of data DN path information, such as: the introduction of the path attribute information file in the file.

303、根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 303, based on the metadata of each data block, generates the metadata information for each data block recovery.

可选的,上述数据块的元数据恢复信息可以包括该数据块的元数据,例如:包括目标文件的属性信息。 Optionally, the metadata of the data block restoration information may include metadata of the data block, for example: the attribute information includes a target file.

304、向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述目标文件的元数据的恢复。 304, to the at least one of the plurality of DN DN transmitting the metadata to restore information, restore the metadata information for the at least one DN is saved and the NN abnormal then sent to the said NN recover the target file of metadata.

由于上述元数据恢复信息被一个或者多个DN保存,这样可以防止单个DN出现异常后,导致该元数据恢复信息丢失。 Because of the above metadata restore one or more DN information is saved so that you can prevent the single DN abnormal, leading to the recovery of metadata information is lost.

可选的,上述从所述NN获取所述多个数据块的元数据可以包括: Optionally, the acquired data of said plurality of blocks from the NN metadata may include:

依次接收所述NN发送的所述每个数据块的元数据。 Sequentially receiving said transmitted NN metadata for each data block.

例如:本端可以是每次向NN申请一个数据块,这样NN就会为目标文件分配一个数据块,以及生成该数据块的元数据,并将该数据块的元数据返回到上述本端。 For example: The client can apply each time a block of data to the NN, and NN will be assigned a data block as the target file, and generate metadata of the data block, and returns the metadata of the data block to the end of the present. 当上述数据块被写完时,而目标文件还没有被写完,本端再向NN申请另一个数据块,NN就会为目标文件分配另一个数据块,以及生成该数据块的元数据,并将该数据块的元数据返回到上述本端。 When said data block is written, and the target file has not been finished, the local application of another data block again NN, NN will allocate another block of data for the target file, and generate metadata of the data block, and returns the metadata of the data block to the end of the present. 当上述数据块被写完时,而目标文件还没有被写完,继续向NN申请数据块,直到目标文件被写完。 When said data block is written, and the target file has not been finished, continue to apply to the NN block until the target file is written. 从而步骤303可以是本端每获取一个数据块的元数据,再生成该数据块的元数据恢复信息,步骤304再将该数据块的元数据恢复信息发送至上述至少一个DN。 Thus step 303 may obtain metadata for each of the end of a block, and then generate the metadata of the data block recovery information, and then step 304 the metadata block recovery information is transmitted to said at least one DN.

可选的,上述从所述NN获取所述多个数据块的元数据可以包括: Optionally, the acquired data of said plurality of blocks from the NN metadata may include:

接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 NN-time receiving the request metadata for all data block to be transmitted according to the write target file.

例如:本端可以是一次性向NN申请多个数据块,这样NN就会为目标文件分配多个数据块,以及生成这些数据块的元数据,并将这些数据块的元数据返回到上述本端。 For example: The end may be a one-time application to NN plurality of data blocks, so that NN will be assigned a plurality of data blocks for the target file, and generate metadata for these data blocks, and returns metadata for these data blocks to the end of the present . 从而步骤303可以是一次性生成多个数据块的元数据恢复信息,步骤304再将这多个数据块的元数据恢复信息发送至上述至少一个DN。 Thus step 303 may be disposable generate metadata information blocks to recover a plurality of data, then in step 304 the metadata of the plurality of data blocks to recover said at least one message sent to the DN.

可选的,上述接收的元数据至少包括第一数据块的元数据和第二数据块的元数据,即目标文件至少被划分为第一数据块和第二数据块。 Optionally, the received metadata comprises metadata for at least a first data block and second data block of metadata, i.e., the target file is divided into at least a first data block and second data block.

其中,上述生成的所述第一数据块的元数据恢复信息可以包括: Wherein said metadata generated by the first data blocks to recover information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息。 Attribute information of the target file and the first data block in the target file location information.

其中,目标文件的属性信息可以包括: Wherein the target file attribute information may include:

HDFS系统的版本信息、文件名、文件的副本数、文件的每个数据块大小、文件占据的空间的限制信息和用户权限信息。 HDFS system version information, file name, file number of copies of each data block size of the file, the file space occupied by the restriction information and user rights information.

上述第一数据块在所述目标文件中的位置信息可以是指,第一数据块的目标文件的数据块序号,例如:第一数据块可以是目标文件的序号为1的数据块,其中,序号为1的数据块可以是目标文件中时间戳最早的一个数据块;上述第一数据块在所述目标文件中的位置信息可以包括: Position information of the first data block in the object file can refer to the data block number of the first data block of the target file, for example: a first data block may be a number of the target file is a block of data, wherein, No. 1 data blocks may be a target file timestamp of the oldest data block; the first data block in the target file location information may include:

数据块序号和文件结束数据块信息。 Block number and the end of file data block information.

上述生成的所述第二数据块的元数据恢复信息可以包括: Generating said metadata of said data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

上述第二数据块在所述目标文件中的位置信息可以包括: Location information of the second block in the target file may include:

数据块序号和文件结束数据块信息。 Block number and the end of file data block information.

另外,第二数据块的元数据恢复信息还可以包括HDFS系统的版本信息和文件名,这样就可以清楚地知道第二数据块的元数据恢复信息是用于恢复哪个文件的数据块,以及该文件的HDFS系统的版本信息。 In addition, the meta data of the second block of data recovery information also may include version information and the name of HDFS file system so that you can clearly know the metadata of the second data block to recover data block information which is used to recover files, and file version information HDFS system.

例如:如图4a所示,第一数据块的元数据恢复信息可以包括: For example: as shown, the metadata data blocks to recover the first information may include, as shown in Figure 4a:

HDFS系统的版本信息、文件名、文件的副本数、文件的每个数据块大小、文件占据的空间的限制信息、用户权限信息、数据块序号和文件结束数据块信息。 HDFS system version information, file name, file number of copies of each data block size of the file, the file space occupied by the restriction information, user rights information, data and files ending block number data block information.

如图4b所示,第一数据块的元数据恢复信息可以包括: 4b, the metadata data blocks to recover the first information may include:

HDFS系统的版本信息、文件名、数据块序号和文件结束数据块信息。 HDFS system version information, file name, block number and the end of file data block information.

其中,文件结束数据块信息可以是用当前数据块是否为结束Block的信息表示,例如:上述第一数据块的元数据恢复信息中的文件结束数据块信息可以用假(false)表示;当上述目标文件只包括两个数据块时,那么,上述第一数据块的元数据恢复信息中的文件结束数据块信息可以用真(true)表 示;当上述目标文件包括三个或者三个以上的数据块时,上述第一数据块的元数据恢复信息和上述第二数据块的元数据恢复信息中的文件结束数据块信息可以用假(false)表示;即该实施方式中,只有目标文件的最后一个数据块对应的文件结束数据块信息才用真(true)表示,其余的都用假(false)表示,从而可以根据文件结束数据块信息可以确定目标文件包括的数据块的个数。 Among them, the end of file data block information may be whether or not to end the Block information indicated by the current data block, for example: the metadata of the first data block in the end of file recovery information data block information can be represented by a false (false); When the above target file includes only two data blocks, then the meta-data of the first data block in the end of file recovery information data block information can be real (true) representation; these objectives documents include three or more data when when the blocks, the metadata of the first data block recovery information and metadata of the second data block in the end of file recovery information data block information can be represented by a false (false); that is, in this embodiment, only the final target file a data block corresponding to the end of file data block information only with real (true) that the rest are used false (false) representation, which can end according to the file number of data blocks in the data block information can be included in determining the target file.

另外,由于一个文件内各数据块的元数据都可以包括文件的属性信息,而文件的属性信息是相同的,例如:文件占据的空间的限制信息、用户权限信息、数据块的大小信息和副本数等信息是相同的,这样就只需要在其中一个数据块的元数据恢复信息中写入这些信息即可,从而可以节省其他DN的存储空间。 In addition, since the meta-data blocks within a file may include a file attribute information, the attribute information file is the same, for example: restriction information, user rights information, size information, and a copy of the file space occupied by the data block number and other information is the same, so you only need one data block metadata recovery information is written to the information, thus saving other DN storage space.

另外,上述仅举例了所述第一数据块和第二数据块的元数据恢复信息的多种可选的实施方式,本实施例中对此不作限定,例如:第一数据块的元数据恢复信息也可以包括: Further, the above-described example only, the first metadata and the second data block of data blocks to recover a variety of optional information embodiments, this embodiment will not be limited to this, for example: a first data block of the metadata recovery information may also include:

HDFS系统的版本信息、文件名、文件的副本数、文件的访问时间、文件的数据块的块数、文件占据的空间的限制信息、用户权限信息、数据块标识、数据块的大小和数据块时间戳。 The number of blocks of data block version information HDFS systems, file names, access times the number of copies the file, file, file, restriction information, user rights information file space occupied by the data block identification data block size and data blocks timestamp.

可选的,本方法中每获取NN发送的数据块的元数据之前,都可以是向NN发送数据块的申请请求,NN接收到该请求后,就可以为目标文件分配数据块,以及存储该数据块的DN。 Alternatively, the present method Per obtain metadata transmitted data block NN, may apply for sending request data blocks to NN, NN after receiving the request, we can target file allocation data block, and stores the DN data block. 另外,在步骤301之前,所述方法还可以包括以下步骤: Further, prior to the step 301, the method may further comprise the steps of:

创建目标文件,并生成该目标文件的概要信息。 Create the target file, and generates summary information of the target file. 例如:生成该文件的文件名等信息。 For example: generates the file name and other information.

可选的,上述方法可以应用于HDFS的客户端(Client),即该客户端可以实现上述方法。 Alternatively, the above method may be applied to HDFS client (Client), that is, the client can achieve the above methods.

上述技术方案中,接收对目标文件的写操作请求,从NN获取多个数据块的元数据,再根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息,从而可以向所述多个DN中的至少一个DN发送所述元数据恢复信息。 The technical proposal, the reception of the target file write request, obtain metadata from a plurality of data blocks NN, then according to the metadata for each data block to generate the metadata for each block of data recovery information, which can transmit at least a DN recover the metadata information to the plurality of the DN. 这样元数据恢复信息可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 Such metadata information can be used by the recovery of at least one DN to save, and then sent to the abnormal NN recover the metadata in the NN. 从而当NN出现异常后,可以及时恢复NN在出现异常时生成的元数据。 So that when the abnormal NN, NN can be restored in the event of abnormal generate metadata in a timely manner.

请参阅图5,图5是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图5所示,包括以下步骤: Refer to FIG. 5, FIG. 5 is a flowchart of another embodiment of the present invention is provided in Example metadata schematic recovery method, shown in Figure 5, comprising the steps of:

501、接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.。 501, metadata receiving client sends the target file recovery information, and storing the recovery information metadata, wherein the metadata information is restored to the client at the time of receiving the target file write request from NN the metadata acquired target file, and recover the metadata information based on the metadata generated ..

其中,上述目标文件所述目标文件保存在多个DN上,且上述元数据包括所述多个DN的信息。 Wherein the target file to the target file is saved on multiple DN, and said metadata includes the plurality of DN information.

可选的,步骤501可以是将元数据恢复信息存储在本地,或者将元数据恢复信息存储第三方面存储设备上。 Alternatively, step 501 may be a meta data recovery information is stored locally, or metadata information is stored on a third aspect of the recovery storage devices.

502、在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 502, after the NN abnormal, sending the information to the meta data recovery NN, metadata restore the recovery information is used for the NN metadata.

可选的,本方法可以应用于HDFS的DN。 Alternatively, the method can be applied to HDFS's DN.

上述技术方案中,接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息;在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 The technical proposal, meta data reception sent by the client object file recovery information, and save the meta data recovery information; after abnormal in the NN, send the information to the meta-data recovery NN, meta-data recovery the information is used for recovery of the NN metadata. 从而可以实现恢复NN出异常时的元数据。 Thereby enabling restoring metadata NN when an abnormality.

请参阅图6,图6是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图6所示,包括以下步骤: Refer to FIG. 6, FIG. 6 is a flowchart of another embodiment of the present invention is provided in Example metadata schematic recovery method, shown in Figure 6, comprising the steps of:

601、接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,所述客户端根据该数据块的元数据生成的元数据恢复信息.。 601, the metadata of the data block receiving the target file recovery information sent by the client, and storing said recovery information metadata, wherein the metadata information is restored in the client receiving the target document when the write operation request, the client information based on the recovery data block to the metadata generated by the metadata ..

其中,上述目标文件被划分为多个数据块,且上述每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 Wherein said target file is divided into a plurality of data blocks, and said metadata about each data block contains the DN information and the information of each said data blocks of each data block resides.

可选的,步骤601接收到元数据恢复信息可以是一个或者多个数据块的元数据恢复信息。 Optionally, the step 601 receives the metadata recovery information may be one or a plurality of data blocks of the metadata recovery information. 例如:目标文件被划分为多个数据块,其中,执行本方法的DN可以是保存其中一个或者多个数据块的元数据恢复信息。 For example: the target file is divided into a plurality of data blocks, wherein the method is performed DN may be stored one or more data blocks to recover the metadata information.

可选的,步骤601可以是将上述元数据恢复信息保存在一个特定文件内,例如:保存在命名为:blk_blockId_genstamp.blockmeta的文件内,其中,该文件的名称可以包括数据块的标识,且该文件名称还可以包括数据块的时间戳,例如:DN在接收到数据块的元数据恢复信息的同时,还可以接收到客户端发送的数据的标识和数据块的时间戳。 Alternatively, step 601 may be the above-mentioned metadata recovery information saved in a specific file, for example: Save the name: within blk_blockId_genstamp.blockmeta documents, which may include the name of the file identification data block, and the file names can also include a time stamp data block, for example: DN received in the metadata block recovery information also can receive identification and time stamp data blocks sent by the client data. 另外,该文件内还可以将客户端在上述数据块中写入的该目标文件的数据保存至另一个特定的文件内,例如:保存在命名为:blk_blockId的文件内。 Further, that document can also be a client in the data block stored in the data written to the destination file to another within a particular file, for example: Save the name: within blk_blockId files. 当客户端还写入了上述数据块的循环冗余校验码(Cyclic Redundancy Check,CRC),本方法还可以将该CRC保存用于保存上述元数据恢复信息的特定文件内,即将上述元数据恢复信息和CRC存储在同一个特定文件内,当然,本方法还可以将CRC存储在一个专门用于存储CRC的特定文件内。 When the client has written the data block cyclic redundancy check code (Cyclic Redundancy Check, CRC), the method can also save the CRC said metadata stored within the recovery information for a specific file, said the upcoming metadata recovery and CRC information stored in a particular file within the same, of course, the present method may also be stored in a dedicated CRC to the CRC stored in a specific file. 另外,上述两个特定文件可以是接收到上述数据信息之前设置好的,例如:接收到客户端发送的写请求时,生成上述两个特定文件。 Further, the above-described two specific file may be provided to the data received before the good information, for example: when receiving the write request sent by the client, generating the two specific file. 由于步骤601将上述数据信息以文件的形式进行存储,这 样可以实现数据仓库工(例如:Hive)功能,或者Spark功能,其中,Spark是一个基于内存计算的开源的集群计算系统,目的是让数据分析更加快速。 Since the step 601 of the data information in the form of a file is stored, so you can achieve a data warehouse workers (for example: Hive) function, or the Spark function, which, Spark is an open source cluster computing system based on in-memory computing, the purpose is to make data analysis more quickly.

另外,上述元数据恢复信息还可以是预定义的数据格式进行保存,例如:以HBase的数据格式进行保存,其中,HBase是一个分布式的、面向列的开源数据库。 Further, the meta-data recovery information can also be predefined data format to save, for example: HBase data format to be saved, which, HBase is a distributed, column-oriented open-source database.

可选的,在获取到上述元数据恢复信息时,还可以在内存中保存上述元数据恢复信息,这样可以更快速地发送该元数据恢复信息。 Alternatively, when obtaining the above metadata recovery information, you can also save metadata in memory, said recovery information, so you can more quickly transmit the meta data recovery information. 可以直接将内存的元数据恢复信息上传到NN。 Metadata can be directly uploaded to the memory recovery information NN. 另外,当实现本方法的DN重新启动后,可以扫描本地存储的元数据恢复信息,再将该元数据恢复信息上传给NN。 In addition, when carrying out the method of DN restart can be scanned locally stored metadata recovery information, and then restore the metadata information is uploaded to the NN.

602、在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述目标文件的元数据的恢复。 602, after the NN abnormal, sending the information to the meta-data recovery NN, meta-data recovery recovery information is used for the NN the target document metadata.

例如:DN在NN重新启动时,会接收到该NN发送的通知消息,该通知消息用于请求发送上述数据信息,这样就可以响应消息通知消息,以返回上述元数据恢复信息。 For example: DN when NN restart, will receive a notification message that NN sent the notification message for requesting to transmit the data, so that you can respond to the message notification message, to return to the above-described metadata recovery information.

可选的,所述接收所述客户端发送的所述目标文件的数据块的元数据恢复信息包括: Metadata optional data block, said receiving the client sends the destination file recovery information includes:

依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata sequentially receiving a plurality of data blocks sent by the client end of the object file recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information sent at once.

可选的,所述接收的元数据恢复信息至少可以包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Optionally, the recovery information received metadata may include at least a first data block of metadata and metadata information recovery data blocks to recover a second information, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息可以包括: The metadata data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

另外,本实施例中,当所述生成的元数据出现异常后,可以是向NN发送目标文件的所有数据块的元数据恢复信息。 Further, in this embodiment, when the generated metadata is abnormal, the target file can be sent to the NN metadata for all data blocks recovery information. 例如:当DN接收到NN异常重启的通知消息后,响应该通知消息扫描本地或者内存中所有数据块的元数据恢复信息,并向NN上传所有数据块的元数据恢复信息。 For example: When the DN receives the notification message NN abnormal restart, in response to the notification message scan local or metadata in memory of all data blocks recovery information, and NN upload metadata for all data blocks recovery information.

可选的,上述方法可以应用于HDFS的DN,即该DN可以实现上述方法。 Alternatively, the above method may be applied to the HDFS DN, that is, the DN can achieve the above methods.

上述技术方案中,接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,并保存所述元数据恢复信息,在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 After the technical proposal, the metadata receiving the data blocks sent by the client of the target file recovery information, save and restore the metadata information, an exception in the NN, sending the dollar to the NN data recovery information, meta data recovery recovery information is used for the NN the metadata. 从而可以及时恢复NN在出现异常时生成的元数据。 Thus it can be restored in the event of abnormal NN metadata generated in a timely manner.

请参阅图7,图7是本发明实施例提供的另一种元数据恢复方法的流程示意图,如图7所示,包括以下步骤: Refer to FIG. 7, FIG. 7 is a flowchart of another embodiment of the present invention is provided in case of the metadata recovery process schematic shown in Figure 7, comprising the steps of:

701、接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上。 701, the receiving client sent to the target file write request, wherein the target file is saved on multiple DN.

702、生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息。 702, to generate the target file's metadata and sends the metadata to the client, wherein the plurality of metadata containing information DN.

可选的,步骤702可以是接收到上述写操作请求后,为上述目标文件分配所述多个DN,再生成上述目标文件的元数据。 Alternatively, after step 702 may be receiving the request to write said allocation of said plurality of object files to DN, and then generates the meta data of the target file.

703、当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至 所述至少一个DN进行保存的元数据恢复信息。 703, when the end of the abnormal, metadata receiving the plurality of DN DN send at least one message of recovery, restoration and recovery information of the metadata according to the metadata, wherein the metadata information is restored the client-generated based on the metadata, sent by the client to the at least one DN to save metadata recovery information.

这样当本端,即NN出现异常时,可以接收到上述至少一个DN发送的元数据恢复信息,从而可以恢复出上述目标文件的元数据。 So that when this end, namely NN abnormal, can receive said at least one metadata DN send recovery information, which can recover the metadata above the target file.

可选的,上述目标文件可以被划分为多个数据块,所述多个数据块被保存在所述多个DN上; Alternatively, the above-described object file can be divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,上述生成所述目标文件的元数据,并向所述客户端发送所述元数据可以包括: Then, the generation of the target file's metadata, the client sends to the metadata may include:

生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 Generating the plurality of data blocks of metadata to the client sends the metadata of the plurality of data blocks, the metadata for each data block contains information about each of the data blocks and each data block is located the DN information.

上述接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,可以包括: Transmitting at least one metadata DN restoring said plurality of receiving the information in the DN, and restore the recovery information metadata according to the metadata, may include:

接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Receiving said plurality of data blocks of said at least one of the plurality of DN DN metadata transmitted recovery information and the recovery information recovery target file based on metadata of said plurality of data blocks of metadata.

这样可以实现根据多个数据块的元数据恢复信息恢复目标文件的元数据。 This allows to recover information based on metadata plurality of data blocks to restore the target file's metadata.

可选的,向所述客户端发送所述多个数据块的元数据包括: Optionally, the client sends the plurality of data blocks of said metadata includes:

依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially sends to the client the metadata for the plurality of data blocks; or request a one-time write operation according to the target file metadata for all data blocks to send to the client.

可选的,上述多个数据块至少可以包括第一数据块和第二数据块,其中: Alternatively, said plurality of data blocks may include at least a first data block and second data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息可以包括: The metadata data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

需要说明的是,本实施例中针对元数据和元数据恢复信息的定义可以参考图2或者图3所示的实施例,此处不作重复说明。 It should be noted, for example, to recover information for metadata and metadata definitions can refer to the present embodiment shown in FIG. 2 or 3 examples here will not be repeated.

可选的,上述方法可以应用于HDFS内的NN。 Alternatively, the above-described method can be applied within the NN HDFS.

上述技术方案中,接收客户端发送的对目标文件进行写操作请求,生成所述目标文件的元数据,并向所述客户端发送所述元数据;这样当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据。 The technical proposal, the receiver sent by the client to the target file write request, generating the target file metadata to the client sends the metadata; so that when the end of the abnormal receiving the metadata for at least one of the plurality of DN DN recover transmitted information, and to restore the recovery information metadata according to the metadata. 从而可以恢复NN出异常时的元数据。 NN which can restore the metadata when an abnormality.

下面为本发明装置实施例,本发明装置实施例用于执行本发明方法实施例一至五实现的方法,为了便于说明,仅示出了与本发明实施例相关的部分,具体技术细节未揭示的,请参照本发明实施例一、实施例二、实施例三、实施例四和实施例五。 The following embodiment of the present invention apparatus of the present invention apparatus embodiment of the present invention a method for the implementation of the method to achieve the implementation of one to five cases, for ease of illustration, showing only the relevant parts of the embodiment of the present invention, the specific technical details not disclosed refer to an embodiment of the invention, Example II Example III Example IV and Example V.

请参阅图8,图8是本发明实施例提供的一种客户端的结构示意图,如图8所示,包括:接收单元81、获取单元82、生成单元83和发送单元84,其中: Refer to FIG. 8, FIG. 8 is a block schematic of one embodiment of the present invention to provide the customer's side, as shown in FIG. 8, comprising: a receiving unit 81, acquisition unit 82, a transmission unit generation unit 83 and 84, wherein:

所述接收单元81,用于接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上。 The receiving unit 81 for receiving the target file write request, wherein the target file is saved on multiple DN.

所述获取单元82,用于从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息。 The acquisition unit 82 for acquiring the target file from NN metadata, the target file's metadata contains information about the plurality of DN.

所述生成单元83,用于根据所述目标文件的元数据生成元数据恢复信 息。 The generating unit 83 for generating metadata based on metadata information to restore the target file.

所述发送单元84,用于向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 The sending unit 84, to be used in at least one of the plurality of DN DN transmitting the metadata to restore information, restore the metadata information for the at least one DN is saved, and appears in the NN abnormal then sent to the NN to recover the metadata.

在另一个实施例中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In another embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

则,获取单元82可以用于从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; The acquisition unit 82 may be used for obtaining the plurality of data blocks of metadata from the NN, the metadata of each data block contains the DN information and the information for each of said data blocks of each data block is located;

生成单元83可以用于根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 Generator 83 may be used in accordance with the metadata of each data block, each data block generates the metadata recovery information.

该实施方式中,可以实现生成每个数据块的元数据恢复信息,以使NN根据多个数据块的元数据恢复信息恢复目标文件的元数据。 This embodiment can be implemented to generate metadata for each block of data recovery information to enable NN recover information based on metadata plurality of data blocks to restore the target file's metadata.

可选的,所述获取单元82可以用于依次接收所述NN发送的所述每个数据块的元数据;或者,所述获取单元82可以用于接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 Optionally, the obtaining unit 82 may be configured to sequentially receive the metadata transmitted NN said each data block; Alternatively, the acquiring unit 82 may be used to receive the NN in accordance with said write target file a one-time operation requests metadata for all data blocks sent.

可选的,上述接收的元数据至少可以包括第一数据块的元数据和第二数据块的元数据,其中: Alternatively, the received metadata may include at least a first data block of metadata and metadata of the second data block, wherein:

所述生成的所述第一数据块的元数据恢复信息可以包括: Said metadata generated data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述生成的所述第二数据块的元数据恢复信息可以包括: Generating said metadata of said data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

可选的,上述客户端可以是HDFS的客户端(Client)。 Alternatively, clients can be said HDFS client (Client).

上述技术方案中,接收对目标文件的写操作请求,再从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息,再向所述多个DN中的至少一个DN发送所述元数据恢复信息。 The technical proposal, the reception of the target file write request, then get metadata from the target file NN, generate metadata based on metadata information to restore the target file, again at least in the more DN a DN transmit the meta data recovery information. 这样该元数据恢复信息就可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 So that the recovery of metadata information can be used to save at least one DN and the NN abnormal then sent to the NN to recover the metadata. 从而可以恢复NN出异常时的元数据。 NN which can restore the metadata when an abnormality.

请参阅图9,图9是本发明实施例提供的一种DN的结构示意图,如图9所示,包括:接收单元91和发送单元92,其中: Refer to FIG. 9, FIG. 9 is a block schematic diagram of an embodiment of the DN provided by the embodiment of the present invention, shown in Figure 9, comprising: a receiving unit 91 and transmitting unit 92, wherein:

接收单元91,用于接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.。 Receiving unit 91, the metadata for the target file sent by the client to receive recovery information, and storing the recovery information metadata, wherein the metadata information is to restore the client receives the target file write request when the target file metadata acquired from NN, and recover the information based on the metadata generated metadata ..

发送单元92,用于在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 Transmission unit 92, after the NN for abnormal, sending the information to the meta data recovery NN, the metadata NN recovery information is used for recovery of the metadata.

在另一个实施例中,所述目标文件被划分为多个数据块,所述接收单元91可以用于接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 In another embodiment, the data in the target file is divided into a plurality of data blocks, the receiving unit 91 may be used for the metadata of the data block receiving the target file recovery information sent by the client, meta data blocks to recover the information to the client based on the metadata information to restore the data block of the metadata.

可选的,接收单元91可以用于依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,所述接收单元91可以用于接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata optional plurality of data blocks, for sequentially receiving unit 91 may receive the client endpoint of the object file recovery information; Alternatively, the receiver 91 may be configured to receive the one-time client metadata transmitted plurality of data blocks of the target file recovery information.

可选的,上述接收的元数据恢复信息至少可以包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Alternatively, said recovery information received metadata may include at least a first data block of metadata and metadata information recovery data blocks to recover a second information, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息可以包括: The metadata data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

可选的,本节点可以是HDFS的DN。 Alternatively, this node can be HDFS's DN.

上述技术方案中,接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息;在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 The technical proposal, meta data reception sent by the client object file recovery information, and save the meta data recovery information; after abnormal in the NN, send the information to the meta-data recovery NN, meta-data recovery the information is used for recovery of the NN metadata. 从而可以实现恢复NN出异常时的元数据。 Thereby enabling restoring metadata NN when an abnormality.

请参阅图10,图10是本发明实施例提供的一种NN的结构示意图,如图10所示,包括:接收单元101、生成单元102和恢复单元103,其中: See Figure 10, Figure 10 presents a schematic diagram of an NN is provided in the embodiment of the present invention, shown in Figure 10, comprising: a receiving unit 101, generating unit 102 and the recovery unit 103, wherein:

所述接收单元101,用于接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上。 The receiving unit 101 for receiving the target document sent by the client write requests, wherein the target file is saved on multiple DN.

所述生成单元102,用于生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息。 The generating unit 102 for generating the target file's metadata and sends the metadata to the client, wherein the plurality of metadata containing information DN.

所述恢复单元103,用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 The recovery unit 103 for, when the local abnormal, metadata receiving the plurality of DN DN send at least one message of recovery, restoration and recovery information of the metadata according to the metadata, wherein the the metadata recovery information is generated by the client according to the metadata sent by the client to the at least one DN to save metadata recovery information.

在另一个实施例中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上; In another embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN;

所述生成单元102可以用于生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的 信息和所述每个数据块所在的DN的信息; The generating unit 102 may be used to generate the plurality of data blocks of metadata, the client sends the metadata to the plurality of the data blocks, each data block of metadata containing information for each data block DN information and the location of each data block;

所述恢复单元103可以用于当本端出现异常时,接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 The recovery unit 103 may be used to this end when abnormal, the metadata for the plurality of data blocks of said plurality of receiving at least one DN DN send recovery information and the meta data of said plurality of blocks data recovery information recovery of the target file's metadata.

可选的,生成单元102可以用于生成所述多个数据块的元数据,并依次向所述客户端发送所述多个数据块的元数据;或者,所述生成单元102可以用于生成所述多个数据块的元数据,并根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Alternatively, the generating unit 102 may be used to generate the plurality of data blocks of metadata, and sequentially transmitting the plurality of data blocks of metadata to the client; Alternatively, the generating unit 102 may be used to generate metadata for the plurality of data blocks, and requests a one-time write operation according to the target file sends metadata for all data blocks to the client.

可选的,上述多个数据块至少可以包括第一数据块和第二数据块,其中: Alternatively, said plurality of data blocks may include at least a first data block and second data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息可以包括: The metadata data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

可选的,上述节点可以是HDFS内的NN。 Alternatively, it said node may be within the NN HDFS.

上述技术方案中,接收客户端发送的对目标文件进行写操作请求,生成所述目标文件的元数据,并向所述客户端发送所述元数据;这样当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据。 The technical proposal, the receiver sent by the client to the target file write request, generating the target file metadata to the client sends the metadata; so that when the end of the abnormal receiving the metadata for at least one of the plurality of DN DN recover transmitted information, and to restore the recovery information metadata according to the metadata. 从而可以恢复NN出异常时的元数据。 NN which can restore the metadata when an abnormality.

请参阅图11,图11是本发明实施例提供的另一种元数据恢复设备的结构示意图,包括至少一个处理器1102(例如CPU),至少一个网络接口1105或者其他通信接口,存储器1106,和至少一个通信总线1103,用于实现这 些装置之间的连接通信。 See Figure 11, Figure 11 is an alternative embodiment of the present invention to provide a structure of metadata embodiment schematic view of the recovery device, comprising at least one processor 1102 (e.g., the CPU), at least one network interface, or other communications interfaces 1105, memory 1106, and at least one communication bus 1103 for enabling communication connection between these devices. 处理器1102用于执行存储器1106中存储的可执行模块,例如计算机程序。 A processor for executing a memory 1102 stores the executable module 1106, e.g., a computer program. 存储器1106可能包含高速随机存取存储器(RAM:Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。 Memory 1106 may include high speed random access memory (RAM: Random Access Memory), it may also include non-volatile memory (non-volatile memory), such as at least one disk storage. 通过至少一个网络接口1105(可以是有线或者无线)实现该系统网关与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。 Through at least one network interface 1105 (which may be wired or wireless) to communicate with a gateway to connect to the system at least one other network element, you can use the Internet, a wide area network, local network, MAN and so on.

在一些实施方式中,存储器1106存储了程序11061,程序11061可以被处理器1102执行,这个程序包括: In some embodiments, memory 1106 stores program 11061, the processor 1102 may be program 11061 executed, the program comprising:

接收对目标文件的写操作请求,其中,所述目标文件保存在多个DN上; Receiving target file write request, wherein the target file is saved on multiple DN;

从NN获取所述目标文件的元数据,所述目标文件的元数据包含所述多个DN的信息; Get the target document from NN metadata, the target file metadata contains the DN of the plurality of information;

根据所述目标文件的元数据生成元数据恢复信息; Metadata is information generated based on the metadata of the target file;

向所述多个DN中的至少一个DN发送所述元数据恢复信息,所述元数据恢复信息用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复。 To the at least one of said plurality of DN DN transmitting the metadata to restore information, restore the metadata information for the at least one DN is saved and the NN abnormal then sent to the NN to recover the metadata.

在另一个实施例中,所述目标文件可以被划分为多个数据块,所述多个数据块可以被保存在所述多个DN上;处理器1102执行的从NN获取所述目标文件的元数据,根据所述目标文件的元数据生成元数据恢复信息的程序,可以包括: In another embodiment, the target file can be divided into a plurality of data blocks, said plurality of data blocks can be stored on the plurality of the DN; processor 1102 to obtain the execution of the object file from NN metadata, data recovery program generator when metadata information of the object file, may include:

从所述NN获取所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息; Get the plurality of metadata blocks from the NN, the metadata for each data block contains information DN information for each data block and said each data block is located;

根据所述每个数据块的元数据,生成所述每个数据块的元数据恢复信息。 The meta data for each of said data blocks, each data block generates the metadata recovery information.

可选的,处理器1102执行的从所述NN获取所述多个数据块的元数据 的程序,可以包括: Alternatively, the processor 1102 to obtain the execution of a plurality of data blocks of the program metadata from the NN, may include:

依次接收所述NN发送的所述每个数据块的元数据;或者,接收所述NN根据所述目标文件的写操作请求一次性发送的所有数据块的元数据。 NN transmitting said sequentially receives the metadata of each data block; or NN receiving the request metadata for all data blocks in one transmission operation according to the write target file.

可选的,所述接收的元数据至少可以包括第一数据块的元数据和第二数据块的元数据,其中: Optionally, the received metadata may include at least a first data block of metadata and metadata of the second data block, wherein:

所述生成的所述第一数据块的元数据恢复信息可以包括: Said metadata generated data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述生成的所述第二数据块的元数据恢复信息可以包括: Generating said metadata of said data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

可选的,上述设备可以是HDFS的客户端(Client)所属的设备。 Alternatively, these devices can be HDFS client (Client) device belongs.

上述技术方案中,接收对目标文件的写操作请求;从NN获取所述目标文件的元数据;根据所述目标文件的元数据生成元数据恢复信息;向所述多个DN中的至少一个DN发送所述元数据恢复信息。 The technical proposal, the reception of the target file write request; acquiring the target file from NN metadata; recovery based on metadata information of the target file to generate the metadata; at least one of said plurality of DN DN to the transmitting the meta data recovery information. 这样元数据恢复信息可以用于被所述至少一个DN进行保存,并在所述NN出现异常后被发送到所述NN进行所述元数据的恢复,从而可以恢复NN出异常时的元数据。 Such metadata information can be used to restore at least one DN is saved and the NN abnormal then sent to the NN to recover the metadata, thereby restoring metadata NN when an abnormality.

请参阅图12,图12是本发明实施例提供的一种DN设备的结构示意图,包括至少一个处理器1202(例如CPU),至少一个网络接口1205或者其他通信接口,存储器1206,和至少一个通信总线1203,用于实现这些装置之间的连接通信。 Refer to FIG. 12, FIG. 12 is a structural diagram DN implement a device provided according to the present invention, it comprises at least one processor 1202 (e.g., the CPU), at least one network interface, or other communications interfaces 1205, memory 1206, and at least one communication bus 1203 for effecting connection between the communication devices. 处理器1202用于执行存储器1206中存储的可执行模块,例如计算机程序。 A processor for executing a memory 1202 stores the executable module 1206, e.g., a computer program. 存储器1206可能包含高速随机存取存储器(RAM:Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。 Memory 1206 may include high speed random access memory (RAM: Random Access Memory), it may also include non-volatile memory (non-volatile memory), such as at least one disk storage. 通过至少一个网络接口1205(可以是有线或者无 线)实现该系统网关与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。 Through at least one network interface 1205 (which may be wired or wireless) for communication gateways connecting the system with at least one other network element, you can use the Internet, a wide area network, local network, MAN and so on.

在一些实施方式中,存储器1206存储了程序12061,程序12061可以被处理器1202执行,这个程序包括: In some embodiments, memory 1206 stores program 12061, the processor 1202 may be program 12061 executed, the program comprising:

接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息,其中,所述元数据恢复信息是所述客户端在接收对目标文件的写操作请求时,从NN获取的所述目标文件的元数据,并根据所述元数据生成的元数据恢复信息.; Receiving target file sent by the client metadata recovery information, and storing the recovery information metadata, wherein the metadata information is restored to the client at the time of receiving the target file write request, obtained from NN the target file's metadata, and recover the information based on metadata generated by the metadata.;

在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 After the NN abnormal, sending the information to the meta data recovery NN, metadata restore the recovery information is used for the NN metadata.

在另一个实施例中,所述目标文件被划分为多个数据块,处理器1202执行的接收客户端发送的目标文件的元数据恢复信息的程序,可以包括: In another embodiment, the target file is divided into a plurality of blocks of data, the client receives metadata processor 1202 to perform the transmission destination file recovery program information may include:

接收所述客户端发送的所述目标文件的数据块的元数据恢复信息,所述数据块的元数据恢复信息为所述客户端根据该数据块的元数据生成的元数据恢复信息。 Receiving the data block sent by the client to the target file metadata recovery information, the metadata of the data block recovery information of the client to recover information from the metadata of the data block to generate metadata.

可选的,处理器1202执行的接收所述客户端发送的所述目标文件的数据块的元数据恢复信息的程序,可以包括: Alternatively, the metadata file data block of the target client receiving the processor 1202 executing a program transmitted recovery information may include:

依次接收所述客户端点发送的所述目标文件的多个数据块的元数据恢复信息;或者,接收所述客户端一次性发送的所述目标文件的多个数据块的元数据恢复信息。 Metadata sequentially receiving a plurality of data blocks sent by the client end of the object file recovery information; or metadata of a plurality of data blocks of the target client receives the file recovery information sent at once.

可选的,所述接收的元数据恢复信息至少可以包括第一数据块的元数据恢复信息和第二数据块的元数据恢复信息,其中: Optionally, the recovery information received metadata may include at least a first data block of metadata and metadata information recovery data blocks to recover a second information, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置 信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息可以包括: The metadata data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

可选的,本节点设备可以是HDFS的DN所属的设备。 Alternatively, this device may be a device node of HDFS DN belongs.

上述技术方案中,接收客户端发送的目标文件的元数据恢复信息,并保存所述元数据恢复信息;在所述NN出现异常后,向所述NN发送所述元数据恢复信息,元数据恢复信息被所述NN用于进行所述元数据的恢复。 The technical proposal, meta data reception sent by the client object file recovery information, and save the meta data recovery information; after abnormal in the NN, send the information to the meta-data recovery NN, meta-data recovery the information is used for recovery of the NN metadata. 从而可以实现恢复NN出异常时的元数据。 Thereby enabling restoring metadata NN when an abnormality.

请参阅图13,图13是本发明实施例提供的一种NN设备的结构示意图,包括至少一个处理器1302(例如CPU),至少一个网络接口1305或者其他通信接口,存储器1306,和至少一个通信总线1303,用于实现这些装置之间的连接通信。 Refer to FIG. 13, FIG. 13 is a schematic NN implement a device provided by the present invention, it comprises at least one processor 1302 (e.g., the CPU), at least one network interface, or other communications interfaces 1305, memory 1306, and at least one communication bus 1303 for effecting connection between the communication devices. 处理器1302用于执行存储器1306中存储的可执行模块,例如计算机程序。 A processor for executing a memory 1302 stores the executable module 1306, e.g., a computer program. 存储器1306可能包含高速随机存取存储器(RAM:Random Access Memory),也可能还包括非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。 Memory 1306 may include high speed random access memory (RAM: Random Access Memory), it may also include non-volatile memory (non-volatile memory), such as at least one disk storage. 通过至少一个网络接口1305(可以是有线或者无线)实现该系统网关与至少一个其他网元之间的通信连接,可以使用互联网,广域网,本地网,城域网等。 Through at least one network interface 1305 (which may be wired or wireless) to communicate with a gateway to connect to the system at least one other network element, you can use the Internet, a wide area network, local network, MAN and so on.

在一些实施方式中,存储器1306存储了程序13061,程序13061可以被处理器1302执行,这个程序包括: In some embodiments, memory 1306 stores program 13061, the processor 1302 may be program 13061 executed, the program comprising:

接收客户端发送的对目标文件进行写操作请求,其中,所述目标文件保存在多个DN上; Receiving client sent to the target file write request, wherein the target file is saved on multiple DN;

生成所述目标文件的元数据,并向所述客户端发送所述元数据,其中,所述元数据包含所述多个DN的信息; Generating the target file's metadata, the client sends to the metadata, wherein the metadata contains the DN of the plurality of information;

当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据,其中,所述元数据恢复信息是所述客户端根据所述元数据生成的,由所述客户端发送至所述至少一个DN进行保存的元数据恢复信息。 When the end of the abnormal, metadata receiving the plurality of DN DN send at least one message of recovery, restoration and recovery information of the metadata according to the metadata, wherein the metadata is the recovery information client-generated based on the metadata, sent by the client to the at least one DN to save metadata recovery information.

在另一个实施例中,所述目标文件被划分为多个数据块,所述多个数据块被保存在所述多个DN上;处理器1302执行的生成所述目标文件的元数据,并向所述客户端发送所述元数据,可以包括: In another embodiment, the target file is divided into a plurality of data blocks, said plurality of data blocks are stored on the plurality of the DN; generating metadata for files of the target processor 1302 for execution, and sends the metadata to the client, may include:

生成所述多个数据块的元数据,并向所述客户端发送所述多个数据块的元数据,每个数据块的元数据包含每个数据块的信息和所述每个数据块所在的DN的信息。 Generating the plurality of data blocks of metadata to the client sends the metadata of the plurality of data blocks, the metadata for each data block contains information about each of the data blocks and each data block is located the DN information.

处理器1302执行的接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据的程序,可以包括: DN metadata receiving the plurality of processors 1302 perform at least one of the recovery information sent DN, and restore the recovery information metadata according to the metadata procedures, comprising:

接收所述多个DN中的至少一个DN发送的所述多个数据块的元数据恢复信息,并根据所述多个数据块的元数据恢复信息恢复所述目标文件的元数据。 Receiving said plurality of data blocks of said at least one of the plurality of DN DN metadata transmitted recovery information and the recovery information recovery target file based on metadata of said plurality of data blocks of metadata.

可选的,处理器1302执行的向所述客户端发送所述多个数据块的元数据的程序,可以包括: Alternatively, the processor 1302 to perform the client sends the plurality of data blocks of the program metadata, may include:

依次向所述客户端发送所述多个数据块的元数据;或者,根据所述目标文件的写操作请求一次性向所述客户端发送所有数据块的元数据。 Sequentially sends to the client the metadata for the plurality of data blocks; or request a one-time write operation according to the target file metadata for all data blocks to send to the client.

可选的,上述多个数据块至少可以包括第一数据块和第二数据块,其中: Alternatively, said plurality of data blocks may include at least a first data block and second data block, wherein:

所述第一数据块的元数据恢复信息可以包括: The metadata data blocks to recover the first information may include:

所述目标文件的属性信息和所述第一数据块在所述目标文件中的位置信息; Attribute information of the target file and the first data block in the target file location information;

所述第二数据块的元数据恢复信息可以包括: The metadata data blocks to recover the second information may include:

所述第二数据块在所述目标文件中的位置信息。 Said second data block location information in the target file.

可选的,上述节点设备可以是HDFS内的NN所属的设备。 Alternatively, said node device may be a device NN belongs within HDFS.

上述技术方案中,接收客户端发送的对目标文件进行写操作请求;生成所述目标文件的元数据,并向所述客户端发送所述元数据;当本端出现异常时,接收所述多个DN中的至少一个DN发送的元数据恢复信息,并根据所述元数据恢复信息恢复所述元数据。 The above technical solution, the target file is sent to the receiving client write request; generating the target file metadata to the client sends the metadata; when the end of the abnormal receiving the multi a metadata DN DN send at least one message of recovery, and recovery information to restore the data based on the metadata element. 从而可以恢复NN出异常时的元数据。 NN which can restore the metadata when an abnormality.

本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,所述的程序可存储于一计算机可读取存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。 Those of ordinary skill will be appreciated that the method to achieve the above embodiments all or part of the process, can be achieved through a computer program instructing relevant hardware, the program may be stored in a computer readable storage medium, the program when executed, the steps of the methods described above as an embodiment. 其中,所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存取存储器(Random Access Memory,简称RAM)等。 Wherein the storage medium may be a magnetic disk, an optical disk, Only Memory (Read-Only Memory, ROM) or random access memory (Random Access Memory, referred to as the RAM).

以上所揭露的仅为本发明较佳实施例而已,当然不能以此来限定本发明之权利范围,因此依本发明权利要求所作的等同变化,仍属本发明所涵盖的范围。 The preferred embodiment disclosed above is only the present invention only, of course, not in order to define the scope of the claimed invention, therefore under this claimed invention requires the equivalent changes made to the scope of the present invention is still covered.

Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
CN102024044A *8 Dec 201020 Apr 2011华为技术有限公司Distributed file system
CN102843403A *23 Jun 201126 Dec 2012盛大计算机(上海)有限公司File processing method based on distributed file system, system, and client
CN103076994A *31 Dec 20121 May 2013北京中科大洋科技发展股份有限公司Method for realizing off-line file writing in SAN (Storage Area Network) shared-file system
CN104202387A *27 Aug 201410 Dec 2014华为技术有限公司Metadata recovery method and related device
US20030220943 *23 May 200227 Nov 2003International Business Machines CorporationRecovery of a single metadata controller failure in a storage area network environment
Classifications
International ClassificationH04L29/08
Cooperative ClassificationH04L29/08
Legal Events
DateCodeEventDescription
13 Apr 2016121Ep: the epo has been informed by wipo that ep was designated in this application
Ref document number: 15836410
Country of ref document: EP
Kind code of ref document: A1
27 Feb 2017NENPNon-entry into the national phase in:
Ref country code: DE
20 Sep 2017122Ep: pct application non-entry in european phase
Ref document number: 15836410
Country of ref document: EP
Kind code of ref document: A1