CN104320401A - Big data storage and access system and method based on distributed file system - Google Patents
Big data storage and access system and method based on distributed file system Download PDFInfo
- Publication number
- CN104320401A CN104320401A CN201410602562.3A CN201410602562A CN104320401A CN 104320401 A CN104320401 A CN 104320401A CN 201410602562 A CN201410602562 A CN 201410602562A CN 104320401 A CN104320401 A CN 104320401A
- Authority
- CN
- China
- Prior art keywords
- server
- meta data
- log
- access
- data server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/561—Adding application-functional data or data for application control, e.g. adding metadata
Abstract
The invention discloses a big data storage and access system and method based on a distributed file system. The big data storage and access system based on the distributed file system comprises the distributed file system which comprises a metadata server, a storage server cluster, an access client and a log server, the metadata server manages and monitors metadata information in the distributed file system and the storage server cluster, the storage server cluster stores data contained in the metadata information in multiple data storage nodes of the storage server cluster, the log server backs up the metadata information in the metadata server and synchronizes operation logs of the metadata server in real time, and the access client is used for selecting standby metadata service processes in the metadata server or in the log server according to the connection state of the metadata server to store the server cluster.
Description
Technical field
The present invention relates to data memory access technical field, be specifically related to a kind of large data memory access system and method based on distributed file system.
Background technology
During based on the Namenode (i.e. management node) of hadoop for the production of environment, store single-point problem.The scheme addressed this problem at present is mainly: the metadata information of Namenode is stored in NFS (Network File System, NFS) on, the main frame at the active and standby process place of Namenode, all as the client of NFS, can access the metadata information that NFS stores respectively.After the host node process of Namenode exits, the metadata store that host process is safeguarded is on NFS, the slave node process of Namenode still normally can access the metadata on NFS, and the slave node process of such Namenode has just taken over the work of the host node process of Namenode completely.
Adopt NFS management document to store, management node service end easily runs into disk bottleneck, bandwidth bottleneck, affects service response speed; Management node service end memory space can not the problem such as dynamic expansion.
The first, management node service end disk easily runs into bottleneck: data are unified to be stored on management node service end disk, and multiple different access client initiates data access request simultaneously, and management node service end disk easily runs into bottleneck, affects performance.
Second, management node service end bandwidth easily runs into bottleneck: data are that unification is stored on management node service end disk, when multiple different access client initiates data access request simultaneously, the data flowing to different access client all will flow out from management node service end network interface card, management node service end bandwidth easily reaches bottleneck, affects performance.
3rd, management node service end memory space can not dynamic expansion: under data can only be stored in management node service end catalogue, need stop share service when management node service end insufficient memory is expanded.Behind extension storage space, again start memory space, the process need of expansion stops upper layer application system.
4th, management node service end single-point problem: data are stored in management node service end, service end generation suspension, machine of delaying, all access client all can not visit again the data in service end, applied business can break down along with service end fault, and business can stop service.
Summary of the invention
Technical problem to be solved by this invention is to provide a kind of large data memory access system and method based on distributed file system, can solve management server single-point problem in large data memory access process.
According to one aspect of the present invention, provide a kind of large data memory access system based on distributed file system, comprise distributed file system, described distributed file system comprises meta data server, storage server cluster, access client and log server;
Described meta data server is used for managing the metadata information in distributed file system and carrying out management and monitoring to described storage server cluster;
Described storage server cluster is used for several data memory nodes according to pre-defined rule, the data comprised in metadata information being stored in storage server cluster;
Described log server is used for carrying out backing up and the Operation Log of real-time synchronization meta data server in interval to schedule to the metadata information in described meta data server;
Described access client is used for the connection status according to meta data server, selects by the Metadata Service process access storage server cluster for subsequent use in meta data server or log server.
According to another aspect of the present invention, provide a kind of large data memory access method based on distributed file system, comprising:
According to pre-defined rule, the data comprised in metadata are stored in several memory nodes of storage server cluster, and by meta data server, management and monitoring are carried out to metadata information and described storage server cluster;
Log server to schedule interval backs up the metadata information in meta data server, and the Operation Log of real-time synchronization meta data server;
Access client sends data access request to meta data server;
According to the connection status of meta data server, described access client is selected by the Metadata Service process access storage server cluster for subsequent use in meta data server or log server.
A kind of large data memory access system and method based on distributed file system provided by the invention, store data on several memory nodes of storage server cluster, solve service end disk bottleneck and the bandwidth bottleneck of prior art, and the memory space of service end is easy to dynamic expansion; Meta data server carries out monitoring management to storage server cluster, log server carries out backup to the metadata of meta data server and its Operation Log synchronous, when meta data server is in abnormality, access client can be conducted interviews to storage server cluster by Metadata Service process for subsequent use in log server, solves management node service end single-point problem of the prior art.
Accompanying drawing explanation
Fig. 1 is a kind of large data memory access system schematic based on distributed file system of the embodiment of the present invention one;
Fig. 2 is a kind of large data memory access method flow diagram based on distributed file system of the embodiment of the present invention two.
Embodiment
Be described principle of the present invention and feature below in conjunction with accompanying drawing, example, only for explaining the present invention, is not intended to limit scope of the present invention.
Embodiment one, a kind of large data memory access system based on distributed file system.Below in conjunction with Fig. 1, the system that the present embodiment provides is described in detail.
In Fig. 1, a kind of large data memory access system based on distributed file system that the present embodiment provides comprises distributed file system, wherein, distributed file system comprises meta data server 101, storage server cluster 102, access client 103 and log server 104.
Wherein, meta data server 101 is mainly used in managing the metadata information in distributed file system and carrying out management and monitoring to storage server cluster 102.
Storage server cluster 102 is mainly used in several data memory nodes according to pre-defined rule, the data comprised in metadata information being stored in storage server cluster.
Concrete, in the present embodiment, storage server cluster 102 has several management node servers, if each management node server all manages individual storage server node, forms storage server cluster 102, carries out distributed storage to large data.
During concrete enforcement, the metadata information in meta data server 101 pairs of distributed file systems manages, and carries out monitoring management to storage server cluster 102.Meta data server 101 carries out cutting according to pre-defined rule to the data comprised in metadata information, data are cut into some pieces, and according to certain strategy, the some blocks of data after cutting are stored on different storage server nodes, such as, can distribute according to the residual memory space of current storage server, data are stored on the larger storage server node of remaining space as far as possible.
Meta data server 101 also carries out management and monitoring to storage server cluster 102, such as monitors the memory data output on each storage server, in order to avoid the data volume that storage server stores crosses the collapse of ambassador's storage server; For another example, the connection status of each storage server is monitored, to understand the connection status of storage server at any time.
In addition, the virtual IP address configuration module in distributed file system is also to meta data server 101 configuration virtual IP, and access client 103 can log in by virtual IP address the access that meta data server 101 pairs of storage server clusters 102 carry out data.Wherein, meta data server managing process and virtual IP address function binding, both store simultaneously, move simultaneously.
Log server 104 is mainly used in carrying out backing up and the Operation Log of real-time synchronization meta data server 101 in interval to schedule to the metadata information in meta data server 101.
Concrete, storage server cluster 102 has carried out distributed storage to large data, and carries out management and monitoring by meta data server 101 pairs of storage server clusters 102, and the metadata information in log server 104 pairs of meta data servers 101 backs up.
Access client 103 is mainly used in sending data access request to meta data server 101 and according to the connection status of meta data server 101, selecting by Metadata Service process for subsequent use in meta data server 101 or log server 104 access storage server cluster.
Concrete, if when access client 103 wishes to conduct interviews to the data in storage server cluster 102, access client 103 can send data access request to meta data server 101.
When meta data server 101 is in line states, access client 103 logs in meta data server 101 by the virtual IP address that virtual IP address configuration module configures, and then carries out the access of data by meta data server 101 pairs of storage server clusters 102.
(the such as suspension when meta data server 101 is in off-line state, delay machine or break down), the virtual IP address of meta data server service processes and configuration is switched to log server 104 start, by the function of Metadata Service process adapter meta data server 101 for subsequent use in log server 104, namely now by Metadata Service process for subsequent use in log server 104, storage server cluster 102 is managed, the virtual IP address that access client 103 still can be configured to meta data server 101 by virtual IP address configuration module logs in log server 104, and then by Metadata Service process for subsequent use in log server 104, storage server cluster 102 is carried out to the access of data.
As previously mentioned, log server 104 can back up the metadata information in meta data server 101 at interval to schedule, and the Operation Log of meeting real-time synchronization meta data server 101.When meta data server 101 is in off-line state, the virtual IP address of meta data server service processes and configuration is switched to log server 104 and starts.Now, the metadata information that Metadata Service process for subsequent use in log server 104 can back up from meta data server 101 according to log server 104 and synchronous Operation Log recover up-to-date metadata information, conduct interviews for access client.Metadata Service process for subsequent use in log server 104 is according to the up-to-date metadata information recovered, load the data in this metadata information recovered, start managing process, whole distributed file system is managed, access client 103 now can log in log server 104 by former virtual IP address, by the Metadata Service process for subsequent use in log server, storage server cluster 102 is carried out to the access of data.
Embodiment two, a kind of large data memory access method based on distributed file system.Below in conjunction with Fig. 2, the method that this enforcement provides is described in detail.
In Fig. 2, S201, according to pre-defined rule, the data comprised in metadata information are stored in several memory nodes of storage server cluster, and by meta data server, management and monitoring are carried out to storage server cluster.
Concrete, in prior art, storage server cluster only has a management node server, store data in storage server cluster, when access client conducts interviews to the data in storage server cluster, all need to carry out data outflow by management node server network interface card, therefore can there is memory space and the bandwidth bottleneck of management node server.In the present embodiment, storage server cluster has several management node servers, if each management node server all manages individual storage server node, forms storage server cluster, carries out distributed storage to large data.
During concrete enforcement, meta data server manages the metadata information in distributed file system, and carries out monitoring management to storage server cluster.Such as, metadata information (comprising file directory tree tissue, attribute maintenance, the record of file operation daily record, the granted access etc. of access client) in the managing distributed file system of meta data server, manage the NameSpace of whole storage system, the data storage and access of coordinating access client and storage server collection cluster, provides the management to name space, access control information, file and the information such as memory block map information, block position.Meta data server carries out cutting according to pre-defined rule to the data comprised in metadata information, data are cut into some pieces, and according to certain strategy, the some blocks of data after cutting are stored on different storage server nodes, such as, can distribute according to the residual memory space of current storage server, data are stored on the larger storage server node of remaining space as far as possible.
Meta data server also carries out management and monitoring to storage server cluster, such as monitors the memory data output on each storage server, in order to avoid the data volume that storage server stores crosses the collapse of ambassador's storage server; For another example, the connection status of each storage server is monitored, to understand the connection status of storage server at any time.
In addition, distributed file system is also to meta data server configuration virtual IP, and access client can log in meta data server by virtual IP address, and then storage server cluster is carried out to the access of data by meta data server.Wherein, meta data server managing process and virtual IP address function binding, both store simultaneously, move simultaneously.
S202, log server to schedule interval back up the metadata information in meta data server, and the Operation Log of real-time synchronization meta data server.
Concrete, above-mentioned steps has carried out distributed storage to large data, and carries out management and monitoring by meta data server to storage server cluster, and this step log server backs up the metadata information in meta data server.During concrete enforcement, log server to schedule interval backs up the metadata information in meta data server, such as, can back up every one hour to the metadata information in meta data server.Log server is to the Operation Log of meta data server also real-time synchronization, such as, access client has carried out renewal rewards theory by meta data server to storage server cluster, Operation Log can be formed, these Operation Logs of log server meeting real-time synchronization meta data server in meta data server.
S203, access client send data access request to meta data server.
Concrete, data are stored in after in storage server cluster according to pre-defined rule, if when access client wishes to conduct interviews to the data in storage server cluster, access client can send data access request to meta data server.
S204, connection status according to meta data server, access client is selected by Metadata Service process for subsequent use in meta data server or log server access storage server cluster.
Concrete, distributed file system can carry out heartbeat detection in real time to meta data server, namely carries out real-time monitoring to the connection status of meta data server.When meta data server is in line states, the virtual IP address that access client is configured by distributed file system logs in meta data server, and then storage server cluster is carried out to the access of data by meta data server.During concrete enforcement, when meta data server receives the data access request of access client, first meta data server can be verified the access rights of access client, if when the access authority verification of access client is passed through, meta data server can match the storage server for access client access according to predetermined routing rule.Such as, several access client send data access request to meta data server simultaneously, meta data server can carry out equally loaded, and what be in admirable proportion by data access pressure is distributed between storage server cluster, in order to avoid cause storage server cluster to collapse.
(the such as suspension when meta data server is in off-line state, delay machine or break down), the virtual IP address of meta data server service processes and configuration is switched to log server startup, by the function of Metadata Service process adapter meta data server for subsequent use in log server, namely now by Metadata Service process for subsequent use in log server, storage server cluster is managed, access client still can log in Metadata Service process for subsequent use in log server by distributed file system to the virtual IP address that meta data server configures, and then by Metadata Service process for subsequent use, storage server cluster is carried out to the access of data.
As previously mentioned, log server can back up the metadata in meta data server at interval to schedule, and the Operation Log of meeting real-time synchronization meta data server.After access client is conducted interviews to storage server cluster by meta data server, Operation Log can be formed in meta data server, such as the data in storage server cluster are upgraded, renewal rewards theory daily record can be formed in meta data server.
When meta data server is in off-line state, the virtual IP address of meta data server service processes and configuration is switched to log server startup.Now, the metadata information that Metadata Service process for subsequent use in log server can back up from meta data server according to log server and synchronous Operation Log recover up-to-date metadata information, load the up-to-date metadata information recovered, conduct interviews for access client.Such as, log server backed up the metadata information in meta data server every 1 hour, and the Operation Log in real-time synchronization meta data server, if now meta data server is in off-line state, Metadata Service process then for subsequent use in log server, according to the metadata information in 1 hour before the meta data server off-line of backup and the Operation Log in this 1 hour, recovers the metadata information in this 1 hour.
Metadata Service process for subsequent use in log server is according to the up-to-date metadata information recovered, load the metadata information that this recovers, start managing process, manage whole distributed file system, access client now can carry out the access of data to storage server cluster by Metadata Service process for subsequent use in log server.
A kind of large data memory access system and method based on distributed file system provided by the invention, by large data according to several memory nodes being stored in storage server cluster after predetermined regular cutting, utilize the feature of distributed file system distributed storage data, data are stored on the different memory nodes of storage server cluster, solve the memory space problem of the server disk of prior art; When access client carries out data access, data flow flows out from different data memory nodes, solves the bandwidth bottleneck of back end, also accelerates the access speed of data; Memory node number in storage server cluster can increase dynamically according to concrete needs or delete, and under the prerequisite not affecting current service, the memory space of storage server cluster can dynamic retractility.
Log server is added in existing distributed file system, the metadata information of management node (alternatively meta data server) and Operation Log in real-time synchronization distributed file system, distributed file system has the switching using HA (High Availability, High Availabitity) to do managing process.When meta data server break down, suspension or delay machine time, HA does the switching of managing process, the service processes of meta data server is switched to Metadata Service process for subsequent use in log server, the function of Metadata Service process adapter meta data server for subsequent use in log server, managing distributed file system, access client can by Metadata Service process for subsequent use in log server access storage server cluster.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1., based on a large data memory access system for distributed file system, it is characterized in that, comprise distributed file system, described distributed file system comprises meta data server, storage server cluster, access client and log server;
Described meta data server is used for managing the metadata information in distributed file system and carrying out management and monitoring to described storage server cluster;
Described storage server cluster is used for several data memory nodes according to pre-defined rule, the data comprised in metadata information being stored in storage server cluster;
Described log server is used for carrying out backing up and the Operation Log of real-time synchronization meta data server in interval to schedule to the metadata information in described meta data server;
Described access client is used for the connection status according to meta data server, selects by the Metadata Service process access storage server cluster for subsequent use in meta data server or log server.
2. large data memory access system as claimed in claim 1, it is characterized in that, described distributed file system also comprises:
Virtual IP address configuration module, for described meta data server configuration virtual IP;
Access client logs in meta data server access storage server cluster by described virtual IP address.
3. large data memory access system as claimed in claim 2, it is characterized in that, described distributed file system also comprises:
Handover module, for when meta data server is in off-line state, switches to log server and starts by meta data server service processes and virtual IP address.
4. large data memory access system as claimed in claim 1, it is characterized in that, described log server also comprises:
Data recovery module, for when meta data server is in off-line state, the metadata information backed up from meta data server according to described log server and synchronous Operation Log, recover up-to-date metadata information.
5. based on a large data memory access method for distributed file system, it is characterized in that, it adopts the arbitrary described large data memory access system based on distributed file system of Claims 1-4, comprising:
Step S201, according to pre-defined rule, the data comprised in metadata information are stored in several memory nodes of storage server cluster, and by meta data server, management and monitoring are carried out to described metadata information and storage server cluster;
Step S202, log server to schedule interval back up the metadata information in meta data server, and the Operation Log of real-time synchronization meta data server;
Step S203, access client send data access request to meta data server;
Step S204, connection status according to meta data server, described access client is selected by the Metadata Service process access storage server cluster for subsequent use in meta data server or log server.
6. large data memory access method as claimed in claim 5, it is characterized in that, described step S201 comprises:
According to pre-defined rule, the data comprised in metadata information are carried out cutting, data after described cutting are stored in several memory nodes of storage server cluster, and by meta data server, management and monitoring are carried out to the memory data output of each memory node of described storage server cluster and presence.
7. large data memory access method as claimed in claim 5, is characterized in that, also comprise:
For meta data server configuration virtual IP, access client logs in described meta data server by virtual IP address and accesses described storage server cluster.
8. large data memory access method as claimed in claim 7, is characterized in that, also comprise before performing step S204:
Heartbeat detection is carried out to meta data server;
Described step S204 comprises:
When meta data server is in line states, access client logs in meta data server access storage server cluster by virtual IP address; When meta data server is in off-line state, access client logs in log server by virtual IP address, by the Metadata Service process access storage server cluster for subsequent use in log server.
9. large data memory access method as claimed in claim 8, is characterized in that, also comprise:
When described meta data server is in off-line state, meta data server service processes and virtual IP address are switched to log server to start, access client logs in log server by virtual IP address, by the Metadata Service process access storage server cluster for subsequent use in log server.
10. large data memory access method as claimed in claim 8, it is characterized in that, described step S204 also comprises:
When meta data server is in off-line state, log server, according to the metadata information backed up from meta data server and synchronous Operation Log, recovers up-to-date metadata information, and access client accesses up-to-date metadata information by log server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410602562.3A CN104320401B (en) | 2014-10-31 | 2014-10-31 | A kind of big data storage based on distributed file system accesses system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410602562.3A CN104320401B (en) | 2014-10-31 | 2014-10-31 | A kind of big data storage based on distributed file system accesses system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104320401A true CN104320401A (en) | 2015-01-28 |
| CN104320401B CN104320401B (en) | 2018-06-22 |
Family
ID=52375571
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410602562.3A Active CN104320401B (en) | 2014-10-31 | 2014-10-31 | A kind of big data storage based on distributed file system accesses system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104320401B (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104618487A (en) * | 2015-02-06 | 2015-05-13 | 杭州华三通信技术有限公司 | Internet protocol storage on-line upgrading method and device |
| CN105049504A (en) * | 2015-07-09 | 2015-11-11 | 国云科技股份有限公司 | Big data transit transmission synchronization and storage method |
| CN105389350A (en) * | 2015-10-28 | 2016-03-09 | 浪潮(北京)电子信息产业有限公司 | Distributed file system metadata information acquisition method |
| CN105468295A (en) * | 2015-11-14 | 2016-04-06 | 华中科技大学 | Safety protection access method and system for realizing object storage |
| CN106648979A (en) * | 2016-12-09 | 2017-05-10 | 曙光信息产业(北京)有限公司 | Backup method and system for distributed real-time file system |
| CN106815286A (en) * | 2016-12-05 | 2017-06-09 | 大唐网络有限公司 | Document handling method |
| CN107070972A (en) * | 2016-12-30 | 2017-08-18 | 中国银联股份有限公司 | A kind of distributed document processing method and processing device |
| CN107229425A (en) * | 2017-06-02 | 2017-10-03 | 浙江宇视科技有限公司 | A kind of date storage method and device |
| CN107357922A (en) * | 2017-07-21 | 2017-11-17 | 郑州云海信息技术有限公司 | A kind of NFS of distributed file system accesses auditing method and system |
| CN107659644A (en) * | 2017-09-25 | 2018-02-02 | 郑州云海信息技术有限公司 | A kind of method, apparatus for accessing storage system, readable storage medium storing program for executing |
| CN108023953A (en) * | 2017-12-04 | 2018-05-11 | 北京小度信息科技有限公司 | The high availability implementation method and device of FTP service |
| CN108259543A (en) * | 2016-12-29 | 2018-07-06 | 广东中科遥感技术有限公司 | Distributed cloud storage database and its be deployed in the method for multiple data centers |
| CN108874319A (en) * | 2018-06-27 | 2018-11-23 | 郑州云海信息技术有限公司 | A kind of metadata updates method, apparatus, equipment and readable storage medium storing program for executing |
| CN109407977A (en) * | 2018-09-25 | 2019-03-01 | 佛山科学技术学院 | A kind of big data distributed storage management method and system |
| CN109783462A (en) * | 2018-12-13 | 2019-05-21 | 创新科存储技术有限公司 | A kind of data access method and device based on distributed file system |
| CN111212111A (en) * | 2019-12-17 | 2020-05-29 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
| CN111400100A (en) * | 2020-03-16 | 2020-07-10 | 北京伟杰东博信息科技有限公司 | Management method and system for distributed software backup |
| CN113225362A (en) * | 2020-02-06 | 2021-08-06 | 北京京东振世信息技术有限公司 | Server cluster system and implementation method thereof |
| CN115242717A (en) * | 2022-09-15 | 2022-10-25 | 四川汉唐云分布式存储技术有限公司 | Centralized storage system based on end-side calculation and control method |
| CN115964353A (en) * | 2023-03-10 | 2023-04-14 | 阿里巴巴（中国）有限公司 | Distributed file system and access metering method thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6173293B1 (en) * | 1998-03-13 | 2001-01-09 | Digital Equipment Corporation | Scalable distributed file system |
| CN101997823A (en) * | 2009-08-17 | 2011-03-30 | 联想(北京)有限公司 | Distributed file system and data access method thereof |
| CN103064927A (en) * | 2012-12-21 | 2013-04-24 | 曙光信息产业(北京)有限公司 | Data access method and device of distributed file system |
| CN103095758A (en) * | 2011-11-01 | 2013-05-08 | 北大方正集团有限公司 | Distributed file system and file data processing method thereof |
| CN103647797A (en) * | 2013-11-15 | 2014-03-19 | 北京邮电大学 | Distributed file system and data access method thereof |
-
2014
- 2014-10-31 CN CN201410602562.3A patent/CN104320401B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6173293B1 (en) * | 1998-03-13 | 2001-01-09 | Digital Equipment Corporation | Scalable distributed file system |
| CN101997823A (en) * | 2009-08-17 | 2011-03-30 | 联想(北京)有限公司 | Distributed file system and data access method thereof |
| CN103095758A (en) * | 2011-11-01 | 2013-05-08 | 北大方正集团有限公司 | Distributed file system and file data processing method thereof |
| CN103064927A (en) * | 2012-12-21 | 2013-04-24 | 曙光信息产业(北京)有限公司 | Data access method and device of distributed file system |
| CN103647797A (en) * | 2013-11-15 | 2014-03-19 | 北京邮电大学 | Distributed file system and data access method thereof |
Cited By (34)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104618487B (en) * | 2015-02-06 | 2018-04-20 | 新华三技术有限公司 | A kind of Internet Protocol stores online upgrade method and device |
| CN104618487A (en) * | 2015-02-06 | 2015-05-13 | 杭州华三通信技术有限公司 | Internet protocol storage on-line upgrading method and device |
| CN105049504A (en) * | 2015-07-09 | 2015-11-11 | 国云科技股份有限公司 | Big data transit transmission synchronization and storage method |
| CN105049504B (en) * | 2015-07-09 | 2019-03-05 | 国云科技股份有限公司 | A kind of big data transfer transmission is synchronous and storage method |
| CN105389350A (en) * | 2015-10-28 | 2016-03-09 | 浪潮(北京)电子信息产业有限公司 | Distributed file system metadata information acquisition method |
| CN105389350B (en) * | 2015-10-28 | 2019-02-15 | 浪潮(北京)电子信息产业有限公司 | A kind of metadata of distributed type file system information acquisition method |
| CN105468295A (en) * | 2015-11-14 | 2016-04-06 | 华中科技大学 | Safety protection access method and system for realizing object storage |
| CN105468295B (en) * | 2015-11-14 | 2019-03-05 | 华中科技大学 | A kind of security protection access method and system for realizing object storage |
| CN106815286A (en) * | 2016-12-05 | 2017-06-09 | 大唐网络有限公司 | Document handling method |
| CN106815286B (en) * | 2016-12-05 | 2020-09-11 | 大唐网络有限公司 | File processing method |
| CN106648979A (en) * | 2016-12-09 | 2017-05-10 | 曙光信息产业(北京)有限公司 | Backup method and system for distributed real-time file system |
| CN108259543B (en) * | 2016-12-29 | 2021-07-06 | 广东中科遥感技术有限公司 | Distributed cloud storage database and method for deploying same in multiple data centers |
| CN108259543A (en) * | 2016-12-29 | 2018-07-06 | 广东中科遥感技术有限公司 | Distributed cloud storage database and its be deployed in the method for multiple data centers |
| CN107070972B (en) * | 2016-12-30 | 2020-02-07 | 中国银联股份有限公司 | Distributed file processing method and device |
| CN107070972A (en) * | 2016-12-30 | 2017-08-18 | 中国银联股份有限公司 | A kind of distributed document processing method and processing device |
| CN107229425B (en) * | 2017-06-02 | 2020-05-19 | 浙江宇视科技有限公司 | Data storage method and device |
| CN107229425A (en) * | 2017-06-02 | 2017-10-03 | 浙江宇视科技有限公司 | A kind of date storage method and device |
| CN107357922A (en) * | 2017-07-21 | 2017-11-17 | 郑州云海信息技术有限公司 | A kind of NFS of distributed file system accesses auditing method and system |
| CN107659644A (en) * | 2017-09-25 | 2018-02-02 | 郑州云海信息技术有限公司 | A kind of method, apparatus for accessing storage system, readable storage medium storing program for executing |
| CN108023953A (en) * | 2017-12-04 | 2018-05-11 | 北京小度信息科技有限公司 | The high availability implementation method and device of FTP service |
| CN108874319A (en) * | 2018-06-27 | 2018-11-23 | 郑州云海信息技术有限公司 | A kind of metadata updates method, apparatus, equipment and readable storage medium storing program for executing |
| CN108874319B (en) * | 2018-06-27 | 2021-06-29 | 郑州云海信息技术有限公司 | Metadata updating method, device, equipment and readable storage medium |
| CN109407977A (en) * | 2018-09-25 | 2019-03-01 | 佛山科学技术学院 | A kind of big data distributed storage management method and system |
| CN109407977B (en) * | 2018-09-25 | 2021-08-31 | 佛山科学技术学院 | Big data distributed storage management method and system |
| CN109783462A (en) * | 2018-12-13 | 2019-05-21 | 创新科存储技术有限公司 | A kind of data access method and device based on distributed file system |
| CN109783462B (en) * | 2018-12-13 | 2021-01-05 | 创新科技术有限公司 | Data access method and device based on distributed file system |
| CN111212111A (en) * | 2019-12-17 | 2020-05-29 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
| CN113225362B (en) * | 2020-02-06 | 2024-04-05 | 北京京东振世信息技术有限公司 | Server cluster system and implementation method thereof |
| CN113225362A (en) * | 2020-02-06 | 2021-08-06 | 北京京东振世信息技术有限公司 | Server cluster system and implementation method thereof |
| CN111400100A (en) * | 2020-03-16 | 2020-07-10 | 北京伟杰东博信息科技有限公司 | Management method and system for distributed software backup |
| CN115242717B (en) * | 2022-09-15 | 2022-12-09 | 四川汉唐云分布式存储技术有限公司 | Centralized storage system based on end-side calculation and control method |
| CN115242717A (en) * | 2022-09-15 | 2022-10-25 | 四川汉唐云分布式存储技术有限公司 | Centralized storage system based on end-side calculation and control method |
| CN115964353A (en) * | 2023-03-10 | 2023-04-14 | 阿里巴巴（中国）有限公司 | Distributed file system and access metering method thereof |
| CN115964353B (en) * | 2023-03-10 | 2023-08-22 | 阿里巴巴（中国）有限公司 | Distributed file system and access metering method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104320401B (en) | 2018-06-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104320401A (en) | Big data storage and access system and method based on distributed file system | |
| US10530855B2 (en) | Lock state synchronization for non-disruptive persistent operation | |
| CN102411637B (en) | Metadata management method of distributed file system | |
| CN103138912B (en) | Method of data synchronization and system | |
| CN110581782B (en) | Disaster tolerance data processing method, device and system | |
| CN104486445A (en) | Distributed extendable resource monitoring system and method based on cloud platform | |
| CN101808127B (en) | Data backup method, system and server | |
| CN104158693A (en) | A method and a system for disaster recovery backup of data service | |
| CN104301142A (en) | Backup method and device for configuration files | |
| CN105335256B (en) | Switch the methods, devices and systems of backup disk in whole machine cabinet server | |
| CN111182067A (en) | Data writing method and device based on interplanetary file system IPFS | |
| CN104580432A (en) | Memcached system, memory cache data provision method and device, memory cache data maintenance method and device as well as cluster maintenance method and device | |
| CN103795801A (en) | Metadata group design method based on real-time application group | |
| CN105262640A (en) | System and method for improving reliability of cloud platform server, and disposition framework of system | |
| CN104104739A (en) | Data synchronization method for metadata service cluster of cloud storage system | |
| CN105069152A (en) | Data processing method and apparatus | |
| CN103986789A (en) | Method for realizing dual redundant of NFS (network file system) nodes in HADOOP HA (home address) cluster based on NFS | |
| US9600487B1 (en) | Self healing and restartable multi-steam data backup | |
| CN102780769A (en) | Cloud computing platform-based disaster recovery storage method | |
| CN103095837A (en) | Method achieving lustre metadata server redundancy | |
| CN202870563U (en) | Distributed comprehensive monitoring system | |
| CN111045602B (en) | Cluster system control method and cluster system | |
| CN104780190A (en) | Data read-write control method and data read-write control device | |
| CN103605616A (en) | Multi-controller cache data consistency guarantee method | |
| CN103414706A (en) | Method and device for managing double-firewall system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |