US20060036896A1 - Method and system for consistent cluster operational data in a server cluster using a quorum of replicas - Google Patents
Method and system for consistent cluster operational data in a server cluster using a quorum of replicas Download PDFInfo
- Publication number
- US20060036896A1 US20060036896A1 US11/203,489 US20348905A US2006036896A1 US 20060036896 A1 US20060036896 A1 US 20060036896A1 US 20348905 A US20348905 A US 20348905A US 2006036896 A1 US2006036896 A1 US 2006036896A1
- Authority
- US
- United States
- Prior art keywords
- replica
- cluster
- node
- members
- quorum
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/18—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
- G06F11/182—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits based on mutual exchange of the output between redundant processing components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
- G06F11/142—Reconfiguring to eliminate the error
- G06F11/1425—Reconfiguring to eliminate the error by reconfiguration of node membership
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
- G06F11/1482—Generic software techniques for error detection or fault masking by means of middleware or OS functionality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/18—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
- G06F11/181—Eliminating the failing redundant component
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2023—Failover techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2035—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant without idle spare hardware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/1658—Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
- G06F11/1662—Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit the resynchronized component or unit being a persistent storage device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
- Physical Water Treatments (AREA)
Abstract
A method and system for increasing server cluster availability by requiring at a minimum only one node and a quorum replica set of replica members to form and operate a cluster. Replica members, independent from the nodes, maintain cluster operational data. A cluster operates when one node possesses a majority of replica members, which ensures that any new or surviving cluster includes consistent cluster operational data via at least one replica member from the immediately prior cluster. Arbitration provides exclusive ownership by one node of the replica members, including at cluster formation, and when the owning node fails. Arbitration uses a fast mutual exclusion algorithm and a reservation mechanism to challenge for and defend the exclusive reservation of each member. A quorum replica set algorithm brings members online and offline with data consistency, including updating unreconciled replica members, and ensures consistent read and update operations.
Description
- This is a continuation of U.S. patent application Ser. No. 09/895,810 filed Jul. 2, 2001, which is a continuation-in-part of U.S. patent application Ser. No. 09/277,450, filed Mar. 26, 1999.
- The invention relates generally to computer network servers, and more particularly to computer servers arranged in a server cluster,
- A server cluster ordinarily is a group of at least two independent servers connected by a network and utilized as a single system. The clustering of servers provides a number of benefits over independent servers. One important benefit is that cluster software, which is run on each of the servers in a cluster, automatically detects application failures or the failure of another server in the cluster. Upon detection of such failures, failed applications and the like can be terminated and restarted on a surviving server.
- Other benefits of clusters include the ability for administrators to inspect the status of cluster resources, and accordingly balance workloads among different servers in the cluster to improve performance. Such manageability also provides administrators with the ability to update one server in a cluster without taking important data and applications offline for the duration of the maintenance activity. As can be appreciated, server clusters are used in critical database management, file and intranet data sharing, messaging, general business applications and the like.
- When operating a server cluster, the cluster operational data (i.e., state) of any prior incarnation of a cluster needs to be known to the subsequent incarnation of a cluster, otherwise critical data may be lost. For example, if a bank's financial transaction data are recorded in one cluster, but a new cluster starts up without the previous cluster's operational data, the financial transactions may be lost. To avoid this, prior clustering technology required that each node (server) of a cluster possess its own replica of the cluster operational data on a private storage thereof, and that a majority of possible nodes (along with their private storage device) of a cluster be operational in order to start and maintain a cluster.
- However, requiring a quorum of nodes has the drawback that a majority of the possible nodes of a cluster have to be operational in order to have a cluster. A recent improvement described in U.S. patent application Ser. No. 08/963,050, entitled “Method and System for Quorum Resource Arbitration in a Server Cluster,” assigned to the same assignee of the present invention, provides the cluster operational data on a single quorum device, typically a storage device, for which cluster nodes arbitrate for exclusive ownership. Because the correct cluster operational data is on the quorum device, a cluster may be formed as long as a node of that cluster has ownership of the quorum device. Also, this ensures that only one unique incarnation of a cluster can exist at any given time, since only one node can exclusively own the quorum device. The single quorum device solution increases cluster availability, since at a minimum, only one node and the quorum device are needed to have an operational cluster. While this is a significant improvement over requiring a majority of nodes to have a cluster, a single quorum device is inherently not reliable, and thus to increase cluster availability, expensive hardware-based solutions are presently employed to provide highly-reliable single quorum device for storage of the operational data. The cost of the highly-reliable storage device is a major portion of the cluster expense.
- Briefly, the present invention provides a method and system wherein at least three storage devices (replica members) are configured to maintain the cluster operational data, and wherein the replica members are independent from any given node. A cluster may operate as long as one node possesses a quorum (e.g., a simple majority) of the configured replica members. For example, in a cluster having three replica members configured, at least two replica members need to be available and controlled by a node to have an operational cluster. Because a replica member can be controlled by only one node at a time, only one unique incarnation of a cluster can exist at any given time, since only one node may possess a quorum of members. The quorum requirement further ensures that a new or surviving cluster has at least one replica member that belonged to the immediately prior cluster and is thus correct with respect to the cluster operational data.
- A quorum arbitration algorithm is provided, by which any number of nodes may arbitrate for exclusive ownership of the replica members (or a single quorum device). The quorum arbitration algorithm ensures that only one node may have possession of the quorum replica set when a cluster is formed, and also enables another node to represent the cluster when a node having exclusive possession of the quorum replica set fails. Arbitration may thus occur when a node first starts up, including when there is no cluster yet established because of a simultaneous startup of the cluster's nodes. Arbitration also occurs when a node loses contact with the owner of the quorum replica set, such as when the owner of the replica set fails or the communication link is broken, as described below.
- In one implementation, arbitration is based on challenging (or defending) for an exclusive reservation of each replica member, and a method for releasing an exclusive reservation is provided. In this implementation, the arbitration process leverages the SCSI command set in order for systems to exclusively reserve the SCSI replica members' resources and break any other system's reservation thereof. A preferred mechanism for breaking a reservation is the SCSI bus reset, while a preferred mechanism for providing orderly mutual exclusion is based on a modified fast mutual exclusion algorithm in combination with the SCSI reserve command. Control of the cluster is achieved when a quorum of replica members is obtained by a node. The algorithm enables any number of nodes to arbitrate for any number of replica members (or for a single quorum device).
- A quorum replica set algorithm is also provided herein to ensure the consistency of data across replica members in the face of replica or node failures. The quorum replica set algorithm provides a database that is both fault tolerant and strongly consistent. The quorum replica set algorithm ensures that changes that were committed in a previous incarnation of the cluster remain committed in the new incarnation of the cluster. Among other things, the quorum replica set algorithm maintains the consistency of data across the replica set as replica members become available (online) or unavailable (offline) to the set. To this end, the quorum replica set algorithm includes a recovery process that determines the most up-to-date replica member from among those in the quorum, and reconciles the states of the available members by propagating the data of that most up-to-date replica member to the other replica members when needed to ensure consistency throughout the replica set. For example, the quorum replica set algorithm propagates the data to update replica members following a cluster failure and restart of the cluster, when a replica member becomes available for use in the replica set (upon the failure and recovery of one or more members), or a change in node ownership of the replica set. The quorum replica set algorithm also handles reads and updates in a manner that maintains consistency, such as by preventing further updates when less than a majority of replica members are successfully written during an update.
- The method and system of the present invention require only a small number of relatively inexpensive components to form a cluster, thereby increasing availability relative to a quorum of nodes solution, while lowering cost and increasing reliability relative to a single quorum device solution.
- Other benefits and advantages will become apparent from the following detailed description when taken in conjunction with the drawings, in which:
-
FIG. 1 is a block diagram representing a computer system into which the present invention may be incorporated; -
FIG. 2 is a representation of various components within the clustering service of a machine; -
FIGS. 3A and 3B are block diagrams representing a server cluster having a plurality of replica members therein for storing cluster operational data in accordance with one aspect of the present invention wherein various cluster components fail over time; -
FIG. 4 is a block diagram representing a server cluster having a plurality of replica members therein for storing cluster operational data in accordance with one aspect of the present invention; -
FIG. 5 is a flow diagram representing general initial steps taken by a node to join a cluster or form a new cluster; -
FIG. 6 is a flow diagram generally representing general logic when forming a cluster in accordance with one aspect of the present invention; -
FIGS. 7A-7C comprise a flow diagram representing general steps taken by a node when operating in a cluster in accordance with one aspect of the present invention; -
FIGS. 8A-8B comprise a flow diagram representing general steps taken by a node to attempt to gain control over a quorum replica set of replica members in accordance with one aspect of the present invention; -
FIGS. 9 and 10 are flow diagrams generally representing steps taken to arbitrate for control of a replica member in accordance with one aspect of the present invention; -
FIG. 11 is a flow diagram generally representing steps taken by a node representing the cluster to defend its ownership of a replica member; -
FIGS. 12A-12D are block diagrams representing changes to logs of quorum replica set members over time, including examples of how the quorum replica set algorithm ensures consistency of replica members in accordance with one aspect of the present invention; -
FIG. 13 is a flow diagram generally representing possible actions taken while a cluster is operating, including actions taken when replica members become available, fail, are read from or are updated; -
FIG. 14 is flow diagram generally representing steps taken by the quorum replica set algorithm when a replica member becomes available for operation in a quorum replica set in accordance with one aspect of the present invention; -
FIGS. 15A-15B comprise a flow diagram generally representing recovery steps taken by the quorum replica set algorithm to make a quorum replica set consistent in accordance with one aspect of the present invention; -
FIGS. 16A-16B comprise a flow diagram generally representing steps taken by the quorum replica set algorithm during recovery to initialize a replica member's log in accordance with one aspect of the present invention; -
FIG. 17 is a flow diagram generally representing steps taken by the quorum replica set algorithm during recovery to reconcile the update logs of the replica members in accordance with one aspect of the present invention; -
FIG. 18 is flow diagram generally representing steps taken by the quorum replica set algorithm when a replica member becomes unavailable for operation during the recovery process in accordance with one aspect of the present invention; -
FIG. 19 is flow diagram generally representing steps taken by the quorum replica set algorithm to read a replica member's log in accordance with one aspect of the present invention; -
FIGS. 20 and 21 are flow diagrams generally representing steps taken by the quorum replica set algorithm to update replica members' logs in accordance with one aspect of the present invention; -
FIG. 22 is flow diagram generally representing steps taken by the quorum replica set algorithm when a replica member becomes unavailable for operation in a quorum replica set in accordance with one aspect of the present invention; -
FIG. 23 is flow diagram generally representing steps taken by the quorum replica set algorithm when a new replica member is added to the configured set of total possible available replica members in accordance with one aspect of the present invention; and -
FIG. 24 is flow diagram generally representing steps taken by the quorum replica set algorithm when a replica member is removed from the configured set of total possible available replica members in accordance with one aspect of the present invention. - Exemplary Operating Environment
-
FIG. 1 and the following discussion are intended to provide a brief general description of a suitable computing environment in which the invention may be implemented. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a personal computer. Generally, program modules include routines, programs, objects, components, data structures and the like that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices. - With reference to
FIG. 1 , an exemplary system for implementing the invention includes a general purpose computing device in the form of a conventionalpersonal computer 20 or the like acting as a node (i.e., system) in a clustering environment. Thecomputer 20 includes aprocessing unit 21, asystem memory 22, and asystem bus 23 that couples various system components including the system memory to theprocessing unit 21. Thesystem bus 23 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read-only memory (ROM) 24 and random access memory (RAM) 25. A basic input/output system 26 (BIOS), containing the basic routines that help to transfer information between elements within thepersonal computer 20, such as during start-up, is stored inROM 24. Thepersonal computer 20 may further include ahard disk drive 27 for reading from and writing to a hard disk, not shown, amagnetic disk drive 28 for reading from or writing to a removablemagnetic disk 29, and anoptical disk drive 30 for reading from or writing to a removableoptical disk 31 such as a CD-ROM or other optical media. Thehard disk drive 27,magnetic disk drive 28, andoptical disk drive 30 are connected to thesystem bus 23 by a harddisk drive interface 32, a magneticdisk drive interface 33, and anoptical drive interface 34, respectively. The drives and their associated computer-readable media provide non-volatile storage of computer readable instructions, data structures, program modules and other data for thepersonal computer 20. Although the exemplary environment described herein employs a hard disk, a removablemagnetic disk 29 and a removableoptical disk 31, it should be appreciated by those skilled in the art that other types of computer readable media which can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read-only memories (ROMs) and the like may also be used in the exemplary operating environment. - A number of program modules may be stored on the hard disk,
magnetic disk 29,optical disk 31,ROM 24 orRAM 25, including an operating system 35 (which may be considered as including or operatively connected to a file system), one ormore application programs 36,other program modules 37 andprogram data 38. A user may enter commands and information into thepersonal computer 20 through input devices such as akeyboard 40 andpointing device 42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner or the like. These and other input devices are often connected to theprocessing unit 21 through aserial port interface 46 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or universal serial bus (USB). Amonitor 47 or other type of display device is also connected to thesystem bus 23 via an interface, such as avideo adapter 48. In addition to themonitor 47, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. - The
personal computer 20 operates in a networked environment using logical connections to one or moreremote computers 49. At least one suchremote computer 49 is another system of a cluster communicating with thepersonal computer system 20 over the networked connection. Otherremote computers 49 may be another personal computer such as a client computer, a server, a router, a network PC, a peer device or other common network system, and typically includes many or all of the elements described above relative to thepersonal computer 20, although only amemory storage device 50 has been illustrated inFIG. 1 . The logical connections depicted inFIG. 1 include a local area network (LAN) 51 and a wide area network (WAN) 52. Such networking environments are commonplace in offices, enterprise-wide computer networks, Intranets and the Internet. Thecomputer system 20 may also be connected to system area networks (SANS, not shown). Other mechanisms suitable for connecting computers to form a cluster include direct connections such as over a serial or parallel cable, as well as wireless connections. When used in a LAN networking environment, as is typical for connecting systems of a cluster, thepersonal computer 20 is connected to thelocal network 51 through a network interface oradapter 53. When used in a WAN networking environment, thepersonal computer 20 typically includes amodem 54 or other means for establishing communications over thewide area network 52, such as the Internet. Themodem 54, which may be internal or external, is connected to thesystem bus 23 via theserial port interface 46. In a networked environment, program modules depicted relative to thepersonal computer 20, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. - A
preferred system 20 further includes ahost adapter 55 or the like which connects thesystem bus 23 to a SCSI (Small Computer Systems Interface)bus 56 for communicating with a quorum replica set 57 (FIG. 3A ) comprising one or more independent, shared persistent memory storage devices, referred to herein as replica members (e.g., 58 1-58 3 ofFIG. 3A ). Other ways of connecting cluster systems to storage devices, including Fibre Channel, are equivalent. Indeed, one alternative way to connect storage devices is via a network connection, as described in U.S. patent application Ser. No. 09/260,194 entitled “Method and System for Remote Access of Computer Devices,” assigned to the assignee of the present invention. - As used herein, a “replica member” is a storage device that is not private to any specific node, but rather is able to be utilized by any node of the cluster at various times. In other words, a replica member can operate in a cluster regardless of which node or nodes are in that particular incarnation thereof. Each replica member may be a simple disk, or some or all of them may be a hardware-based redundant array of devices, although as will become apparent, a benefit of the present invention is that such hardware-based redundancy is unnecessary. Note that any number of replica members (i.e., greater than two in the present invention) may be configured in a given cluster configuration, however for purposes of simplicity only three are shown in
FIG. 3A . In any event, as shown inFIG. 3A , the computer system 20 (FIG. 1 ) may comprise the node 60 1 of acluster 59, while one of the remote computers 49 (FIG. 1 ) may be similarly connected to theSCSI bus 56 and comprise thenode 602, and so on. - Cluster Service Components
-
FIG. 2 provides a representation of cluster service components and their general relationships in each of the nodes 60 1-60 n (FIG. 3A ) of acluster 59. As shown inFIG. 2 , to accomplish cluster creation and to perform other administration of cluster resources, nodes, and the cluster itself, a cluster application programming interface (API) 62 is provided. Applications and clustermanagement administration tools 64 call various interfaces in theAPI 62 using remote procedure invocations through RPC (Remote Procedure Calls) or DCOM (Distributed Component Object Model), whether running in the cluster or on an external system. The various interfaces of theAPI 62 may be considered as being categorized by their association with a particular cluster component, i.e., nodes, resources and the cluster itself. - An administrator typically works with groups, each group being a collection of resources (e.g., cluster application resources, names, addresses and so forth) organized to allow an administrator to combine resources into larger logical units and manage them as a unit. Group operations performed on a group affect all resources contained within that group. Usually a group contains all of the elements needed to run a specific application, and for client systems to connect to the service provided by the application. For example, a group may include an application that depends on a network name, which in turn depends on an Internet Protocol (IP) address, all of which are collected in a single group. In a preferred arrangement, the dependencies of all resources in the group are maintained in a directed acyclic graph, known as a dependency tree. Dependency trees are described in more detail in U.S. patent application Ser. No. 08/963,049 entitled “Method and System for Resource Monitoring of Disparate Resources in a Server Cluster,” assigned to the same assignee as the present invention.
- A
cluster service 66 controls the cluster operation on a server cluster 59 (e.g.,FIG. 3A ), and is preferably implemented as a Windows NT® service. Thecluster service 66 includes anode manager 68, which manages node configuration information and network configuration information (e.g., the paths between nodes 60 1-60 n). Thenode manager 68 operates in conjunction with amembership manager 70, which runs the protocols that determine what cluster membership is when a change (e.g., node failure) occurs. A communications manager 72 (kernel driver) manages communications with other nodes of thecluster 59 via one or more network paths. Thecommunications manager 72 sends periodic messages, called heartbeats, to counterpart components on the other nodes of thecluster 59 to provide a mechanism for detecting that the communications path is good and that the other nodes are operational. Through thecommunications manager 72, thecluster service 66 is essentially in constant communication with the other nodes 60 1-60 n of thecluster 59. In a small cluster, communication is fully connected, i.e., all nodes of thecluster 59 are in direct communication with all other nodes. In a large cluster, direct communication may not be possible or desirable for performance reasons. - Nodes 60 1-60 n in the
cluster 59 have the same view of cluster membership, and in the event that one node detects a communication failure with another node, the detecting node broadcasts a message to nodes of thecluster 59 causing other members to verify their view of the current cluster membership. This is known as a regroup event, during which writes to potentially shared devices are disabled until the membership has stabilized. If a node does not respond, it is removed from thecluster 59 and its active groups are failed over (“pulled”) to one or more active nodes. Note that the failure of thecluster service 66 also causes its locally managed resources to fail. - The
cluster service 66 also includes aconfiguration database manager 76 which implements the functions that maintain a cluster configuration database on local storage devices 98 1-98 n (FIG. 4 ) such as a disk and/or memory, and configuration databases 100 1-100 3 (FIG. 4 ) on each of the replica members 58 1-58 3. The databases 100 1-100 3 maintain cluster operational data, i.e., information about the physical and logical entities in thecluster 59, as described below. In one embodiment, the cluster operational data may be split into core-boot data and cluster configuration data, and is maintained in two cluster databases, as described in the copending U.S. patent application Ser. No. 09/277,503 entitled “Data Distribution in a Server Cluster,” filed on Mar. 26, 1999, assigned to the same assignee as the present invention. As described therein, the core-boot data is stored in a database maintained on quorum replica members, while the cluster configuration data is stored in a database on a higher performance/lower cost storage mechanism such as a mirror set of storage elements. Note that the cluster software is aware that the core-boot data is replicated to multiple storage devices, and that the core-boot data has a log per storage device as described below. However, in such an embodiment, the cluster software views the mirror set storage as a single storage device and is generally not cognizant of the replication (which is maintained at a lower level). Thus, the cluster configuration information is viewed by the cluster software as a single database with a single log. Thedatabase manager 76 may cooperate with counterpart database managers of nodes in thecluster 59 to maintain certain cluster information consistently across thecluster 59. Global updates may be used to ensure the consistency of the cluster database in each of the replica members 58 1-58 3 and nodes 60 1-60 n. - A
logging manager 78 provides a facility that works with thedatabase manager 76 of thecluster service 66 to maintain cluster state information across a situation in which a cluster shuts down and a new cluster is later formed with no nodes necessarily being common to the previous cluster, known as a temporal partition. Thelogging manager 78 operates with the log file, preferably maintained in the replica members 58 1-58 3, to unroll logged state changes when forming a new cluster following a temporal partition. - A
failover manager 80 makes resource/group management decisions and initiates appropriate actions, such as startup, restart and failover. Thefailover manager 80 is responsible for stopping and starting the node's resources, managing resource dependencies, and for initiating failover of groups. - The
failover manager 80 receives resource and node state information from at least oneresource monitor 82 and thenode manager 68, for example, to make decisions about groups. Thefailover manager 80 is responsible for deciding which nodes in thecluster 59 should “own” which groups. Those nodes that own individual groups turn control of the resources within the group over to theirrespective failover managers 80. - An
event processor 83 connects the components of thecluster service 66 via an event notification mechanism. Theevent processor 83 propagates events to and from cluster-aware applications (e.g., 84) and to and from the components within thecluster service 66. Anobject manager 88 maintains various cluster objects. Aglobal update manager 90 operates to provide a global, atomic and consistent update service that is used by other components within thecluster service 66. The global update protocol (GLUP) is used by theglobal update manager 90 to broadcast updates to each node 60 1-60 n in thecluster 59. GLUP generally comprises a standard global update message format, state information maintained in each node, and a set of rules that specify how global update should be processed and what steps should be taken when failures occur. - In general, according to the GLUP protocol, one node (e.g. 60 1 of
FIG. 4 ) serves as a “locker” node. The locker node 60 1 ensures that only one global update is in progress at any given time. With GLUP, a node (e.g., 60 2) wishing to send an update to other nodes first sends a request to the locker node 60 1. When any preceding updates are complete, the locker node 60 1 gives permission for this “sender” node 60 2 to broadcast its update to the other nodes in thecluster 59. In accordance with GLUP, the sender node 60 2 sends the updates, one at a time, to the other nodes in a predetermined GLUP order that is ordinarily based on a unique number assigned to each node. GLUP can be utilized to replicate data to the machines of acluster 59, including at least some of the cluster operational data, as described below. A more detailed discussion of the GLUP protocol is described in the publication entitled “Tandem Systems Review”Volume 1,Number 2, June, 1985 pp. 74-84. - A resource monitor 82 runs in one or more processes that may be part of the
cluster service 66, but are shown herein as being separate from thecluster service 66 and communicating therewith via RPC or the like. The resource monitor 82 monitors the health of one or more resources (e.g., 92 1-92 4) via callbacks thereto. The monitoring and general operation of resources is described in more detail in the aforementioned U.S. patent application Ser. No. 08/963,049. - The resources (e.g., 92 1-92 4) are implemented as one or more Dynamically Linked Libraries (DLLs) loaded into the address space of the
Resource Monitor 82. For example, resource DLLs may include physical disk, logical volume (consisting of one or more physical disks), file and print shares, network addresses and names, generic service or application, and Internet Server service DLLs. The resources 92 1-92 4 run in the system account and are considered privileged code. Resources 92 1-92 4 may be defined to run in separate processes, created by thecluster service 66 when creating resources, or they may be run in a common process. - Resources expose interfaces and properties to the
cluster service 66, and may depend on other resources, with no circular dependencies allowed. If a resource does depend on other resources, the resource is brought online after the resources on which it depends are already online, and is taken offline before those resources. Moreover, each resource has an associated list of nodes in thecluster 59 on which this resource may execute. For example, a disk resource may only be hosted on nodes that are physically connected to the disk. Also associated with each resource is a local restart policy, defining the desired action in the event that the resource cannot continue on the current node. - Nodes 60 1-60 n in the
cluster 59 need to maintain a consistent view of time. A time function suitable for this purpose is available in theWindows® 2000 operating system, however in other implementations one of the nodes may include a resource that implements a time service. - From the point of view of other nodes in the
cluster 59 and management interfaces, nodes in thecluster 59 may be in one of three distinct states, offline, online or paused. These states are visible to other nodes in thecluster 59, and thus may be considered the state of thecluster service 66. When offline, a node is not a filly active member of thecluster 59. The node and itscluster service 66 may or may not be running. When online, a node is a fully active member of thecluster 59, and honors cluster database updates, maintains heartbeats, and can own and run groups. Lastly, a paused node is a fully active member of thecluster 59, and thus honors cluster database updates and maintains heartbeats. Online and paused are treated as equivalent states by most of the cluster software, however, a node that is in the paused state cannot honor requests to take ownership of groups. The paused state is provided to allow certain maintenance to be performed. - Note that after initialization is complete, the external state of the node is offline. To join a
cluster 59, following the restart of a node, thecluster service 66 is started automatically. The node configures and mounts local, non-shared devices. Cluster-wide devices are left offline while booting, because they may be in use by another node. The node tries to communicate over the network with the last known members of thecluster 59. When the node discovers any member of thecluster 59, it performs an authentication sequence wherein the existing cluster node authenticates the newcomer and returns a status of success if authenticated, or fails the request if not. For example, if a node is not recognized as a member or its credentials are invalid, then the request to join thecluster 59 is refused. If successful, the newcomer may be sent an updated copy of the shared database or databases. The joining node may use the one or more databases to find shared resources and to bring them online as needed, and also to find other cluster members. If a cluster is not found during the discovery process, a node will attempt to form its own cluster, by acquiring control of a quorum of the replica devices in accordance with one aspect of the present invention, as described below. - Once online, a node can have groups thereon. A group can be “owned” by only one node at a time, and the individual resources within a group are present on the node that currently owns the group. As a result, at any given instant, different resources within the same group cannot be owned by different nodes across the
cluster 59. Groups can be failed over or moved from one node to another as atomic units. Each group has a cluster-wide policy associated therewith comprising an ordered list of owners. A group fails over to nodes in the listed order. - For example, if a resource (e.g., an application) fails, the
failover manager 80 may choose to restart the resource, or to take the resource offline along with any resources dependent thereon. If thefailover manager 80 takes the resource offline, the group is restarted on another node in thecluster 59, known as pushing the group to another node. A cluster administrator may also manually initiate such a group transfer. Both situations are similar, except that resources are gracefully shutdown for a manually initiated failover, while they are forcefully shut down in the failure case. - When an entire node in the
cluster 59 fails, its groups are pulled from the failed node to another node. This process is similar to pushing a group, but without the shutdown phase on the failed node. To determine what groups were running on the failed node, the nodes maintain group information on each node of thecluster 59 in a database or the like (in-memory or persistent) to track which nodes own which groups. To determine which node should take ownership of which groups, those nodes capable of hosting the groups negotiate among themselves for ownership, based on node capabilities, current load, application feedback and/or the group's node preference list. Once negotiation of a group is complete, all members of thecluster 59 update their databases to properly reflect which nodes own which groups. - When a previously failed node comes back online, the
failover manager 80 decides whether to move some groups back to that node, in an action referred to as failback. To automatically failback, groups require a defined preferred owner. There may be an ordered list of preferred owners in a cluster of more than two nodes. Groups for which the newly online node is the preferred owner are pushed from the current owner to the new node. Protection, in the form of a timing window, is included to control when the failback occurs. - Node Arbitration and Consistent Cluster Operational Data VIA a Quorum of Replicas
- In accordance with one aspect of the present invention, the information needed to form and operate a cluster, i.e., the cluster operational data, is replicated to a quorum replica set 57 of the replica members (e.g., 58 1-58 3 of
FIG. 3A ). Such information generally includes node information, information regarding the replica members 58 1-58 3 of the quorum replica set 57, and other critical information. A node of the cluster (e.g., 60 1) needs to obtain exclusive ownership (control) of a quorum replica set 57 of replica members in order to form and maintain a cluster. Control of a quorum replica set establishes a cluster and guarantees that the cluster incarnation is unique, because only one node can have control over the quorum replica set 57 at any one time. Updates to this operational data are replicated to each member of the quorum replica set 57 by the node that has exclusive ownership thereof. Note that if another node wants to access some information in the quorum replica set 57, it does so through the node that owns the replica set. - To create a new cluster, a system administrator runs a cluster installation utility on a system (node) that then becomes a first configured member of the
cluster 59. For anew cluster 59, a total replica set 106 of replica members is created, each member including a database (e.g., 100 1,FIG. 4 ). As described below, to ensure that each replica member is consistent with the state of the previous cluster, a quorum replica set algorithm is executed to select the most updated replica member of the set, and propagate any needed (logged) information therefrom to other replica members. The administrator then configures any resources that are to be managed by the cluster software, possibly including other storage devices. In general, a first system forms a cluster as generally described below with reference toFIG. 6 . At this time, a cluster exists having a single node (e.g., 60 1), after which an installation procedure may be run to add more nodes and resources. Each added node (e.g., 60 2) receives at least a partial copy of the current cluster operational data, (e.g., the cluster database 100 1). This copy includes the information necessary to identify and access the members of the total replica set 106 and the identity of the other known member nodes of the cluster, (e.g., 60 1-60 n). This information is stored on the added node's local storage, (e.g., 98 2). - More particularly, as shown in
FIG. 5 , beginning atstep 500, a node that has been configured to be part of a cluster, but which is not currently participating in an operational instance of that cluster, first assumes that some instance of the cluster is operational and attempts to join that existing cluster, as described previously. If not successful as determined bystep 502, the node will attempt to form a new instance of the cluster by arbitrating for control of a quorum (e.g., a majority) of the total replica set members, as described below with reference toFIGS. 6-11 . If successful as determined bystep 502, the node joins the existing cluster and performs some work as specified by the cluster, i.e., as set by an administrator, as described below with reference toFIGS. 7A-7C . The node continues to perform work until it is shut down, fails, or some event occurs, such as the node stops communicating with the cluster or a replica member fails, as described below. - In accordance with one aspect of the present invention, to form a cluster when a plurality of replica members are configured, a node has to obtain access to a quorum of the replica members 58 1-58 n, e.g., at least a simple majority of the total configured
replica set 106. As described above, the replica members 58 1-58 3 include the cluster operational data on respective databases 100 1-100 3 (FIG. 4 ). The quorum requirement ensures that at least one replica member is common to the previous cluster, whereby via the common member or members and the quorum replica set algorithm, (described below), the cluster will possess the latest cluster operational data. The quorum further ensures that only one unique cluster may be formed at any given time. As a result, the node owning the quorum replica set possesses the information necessary to properly configure a new cluster following a temporal partition. - By way of example,
FIG. 4 shows two quorum replica sets 57 0 and 57 1 which may be formed from the total number of replica members configured 106, (i.e., three in the present example).Replica Set 0 57 0, represented by the surrounding dashed line, was the prior quorum replica set used by the immediately prior cluster for recording cluster operational data, and included replica members 58 2 and 58 3. Some time later, a new cluster is formed withReplica Set 1 57 1 as the quorum replica set, which, as represented by the surrounding solid line, includes replica members 58 1 and 58 2. Since more than half (two or more in the present example) of the total members configured 106 are required to form a cluster, at least one replica member is common to any previous cluster. In the present example, the replica member 58 2 is common to both replica sets, and thus maintains the correct cluster operational data from the prior cluster. Note that any permutation of the server nodes 60 1-60 n may have been operating in the previous cluster, as long as one node was present. Indeed, a significant benefit of the present invention is that at a minimum, only one node need be operational to form and/or maintain a cluster, which greatly increases cluster availability. In addition, even though multiple replica members (e.g., disks) are used to back up the cluster operational data to provide high availability, only a majority of the replica members is required to be functional in order to operate a cluster. -
FIGS. 3A and 3B show how the present invention increases cluster availability. InFIG. 3A , a cluster is operating with eight total components comprising five nodes 60 1-60 5 and a replica set 57 A having three replica members 58 1-58 3 (out of three total replica members configured to work in the cluster). Some time later, as represented inFIG. 3B , only the node 60 4 has survived, (the crossed-out components indicate failures), along with a modified quorum replica set 57 B comprising a majority two members 58 1, and 58 3 of the three possible replica members. Not only is the cluster capable of operating with a minority of nodes, (only one is needed regardless of the total available), but the cluster functions with a minority of total components (three of at least eight). - In keeping with the invention, any node may form a cluster following a temporal partition, regardless of the number of functioning nodes, since by effectively separating the cluster operational data from the nodes, there is no requirement that a majority of nodes be operational. Thus, for example, in
FIG. 4 , the node 60 3 may have formed thelatest cluster 59 by first having obtained exclusive control (described below) of the replica members 58 1 and 58 2 of the quorum replica set 57 1. To this end, as shown inFIG. 6 , the node attempting to form a cluster first arbitrates (viaFIG. 8A ) for control of a quorum replica set (e.g., 57 1) of replica members from the total replica set 106 configured to operate in the cluster, as described below beginning atFIG. 8A ,step 800. - More particularly, because only one node may have possession of the quorum replica set when a cluster is formed, and also because a node having exclusive possession thereof may fail, there is provided a method for arbitrating for exclusive ownership of the replica members, typically by challenging (or defending) for an exclusive reservation of each member. A method for releasing an exclusive reservation may also be provided. Arbitration may thus occur when a node first starts up, including when there is no cluster yet established because of a simultaneous startup of the cluster's nodes. Arbitration also occurs when a node loses contact with the owner of the quorum replica set, such as when the owner of the replica set fails or the communication link is broken, as described below. Arbitration for and exclusive possession of a single quorum device by two nodes are described in detail in the aforementioned U.S. patent application Ser. No. 08/963,050.
- In accordance with another aspect of the present invention, the arbitration/exclusive ownership process has been extended to accommodate a cluster of more than two nodes. Although the algorithm described herein is capable of arbitrating for control of a replica set with a plurality of members, it should be noted that the multiple node arbitration algorithm is applicable to clusters having a single quorum device as the resource. For example, in such an event, the “majority” can be considered as one member available out of a total configured set of one, and, although some simplification to the algorithm is possible when there is only one device in contention, the general principles are essentially the same.
- In general, to obtain control over the members of the quorum replica set 57 1, an arbitration process leverages a resource reservation mechanism such as the SCSI command set or the like in order for systems to exclusively reserve the (e.g., SCSI) replica members' resources and break any other system's reservation thereof. Control is achieved when a quorum of replica members is obtained by a node. A preferred mechanism for breaking a reservation is the SCSI bus reset, while a preferred mechanism for providing orderly mutual exclusion is based on a modified fast mutual exclusion algorithm in combination with the SCSI reserve command. One such algorithm is generally described in the reference entitled, “A Fast Mutual Exclusion Algorithm,” Leslie Lamport, ACM Transactions on Computer Systems, 5(1), (February 1987), although such an algorithm needs to be modified (among other things) to make it work properly in an asynchronous system such as a cluster.
-
FIGS. 8A and 8B , in combination withFIGS. 9 and 10 , provide general steps for arbitrating for control of a quorum of the members of a replica set. It should be noted thatFIGS. 8A and 8B assume that the identity of at least a quorum of the members of the replica set are known to the nodes performing arbitration, and further, that a total order is imposed on the replica members, and this order is known to the nodes performing arbitration. As described above, such information is written to a node's local storage when the node is joined to the cluster. - Step 800 of
FIG. 8A begins the process for arbitrating for the replica set by initializing some variables, e.g., setting a loop counter (RetryCount) to zero and a delay interval variable equal to an initial value. Similarly,step 802 initializes some additional variables, setting the current member (according to the known ordering) to the first member of the replica set, and zeroing a count that will be used for tracking the number of owned members against the quorum requirement. Step 802 also sets entries in an array that track which members are owned by the node to false, since no members are owned at this time. Step 804 then tests the current member against the order number of the last member in the total replica set, to determine whether arbitration has been attempted on each member in the total set of replica members. At this time, the first member is still the current member, and thus step 804 branches to arbitrate for this current member, as represented in the steps beginning atstep 900 ofFIG. 9 . -
FIG. 9 represents a suitable arbitration process for a single replica member, (e.g., 58 1), although other arbitration mechanisms are possible. The arbitration process ofFIG. 9 generally begins by first determining if a node owns the replica member 58 1, and if so, whether that node is effectively dead (e.g., crashed or paused/operating very slowly, sometimes referred to as comatose). To this end, step 900 ofFIG. 9 first sets a variable, (myseq), for this arbitration that is guaranteed to be unique to this cluster, e.g., the node's cluster-unique identifier in the high bits of the myseq variable plus a current time value in the low bits. Then, atstep 902, the node (e.g., 60 1) attempts to read a variable, y, from a specific location on the current replica member 58 1. - A first possible outcome to the read request is that the read will fail (as detected at step 90 4) because another node (e.g., 60 2) has previously placed (and not released) a reservation on the quorum member 58 1. At this time, there is a possibility that the other node 60 2 that has exclusive control of the quorum replica member 58 1 has stopped functioning properly, and consequently has left the replica member 58 1 in a reserved (locked) state. Note that the nodes 60 1 and 60 2 are not communicating, and thus there is no way for node 60 1 to know why the communication has ceased, e.g., whether the other node 60 2 has crashed or whether the node 60 1 itself has become isolated from the
cluster 59 due to a communication break. Thus, in accordance with another aspect of the present invention, the arbitration process includes a challenge-defense protocol to the ownership of the members of the quorum replica set 57 1 that can shift representation of the cluster from a failed node 60 2 to another node 60 1 that is operational. - To accomplish the challenge portion of the process, if the read failed, at
step 906, the challenging node 60 1 first uses the SCSI bus reset command to break the existing reservation of the quorum replica member 58 1 held by the other node 60 2. Next, after a bus settling time (e.g., two seconds) at step 908, the node 60 1 saves the unique myseq identifier to a local variable old_y and attempts to write the myseq identifier to the y-variable location on the replica member 58 1. Note that the write operation may fail even though the reservation has been broken because another node may have exclusively reserved the replica member 58 1 (via its own arbitration process) between the execution ofsteps step 912, the node 60 1 knows that another node is competing for ownership, whereby the node 60 1 backs off by failing the arbitration and appropriately returning with a “FALSE” success code. Note that the write may also fail if the replica member has failed, in which event it cannot be owned as a quorum member, whereby the “FALSE” return is also appropriate. - However, if the write was successful as determined at
step 912, the arbitration process of the node 60 1 continues to step 914 where the challenging node 60 1 delays for a time interval equal to at least two times a predetermined delta value. As described below, this delay gives a defending node an opportunity to persist its reservation of the replica member 58 1 and defend against the challenge. Because nodes that are not communicating cannot exchange node time information, the delta time interval is a fixed, universal time interval previously known to the nodes in the cluster, at present equal to a three-second arbitration time, and a bus-settling time of two seconds. Note, however that one bus settling time delay was already taken at step 908, and thus step 914 delays for double the arbitration time but only one additional bus settling time, e.g., eight more seconds. After this delay, step 920 again attempts to read the y-variable from the replica member 58 1. - Returning to step 904, if the reading of the y-variable was successful, then no node had a reservation on the replica member 58 1 and the local variable old_y is set to the y-variable (step 916) that was read. However, it is possible that the read was successful because it occurred just after another arbitrating node broke the exclusive reservation of a valid, operational owner. Thus, before giving the node 60 1 exclusive control (ownership) of the replica member 58 1, step 916 branches to step 918 to delay for a period of time sufficient to enable the present exclusive owner, (if there is one), enough time (e.g., the full two-delta time of ten seconds) to defend its exclusive ownership of the current member. After the delay,
step 918 continues to step 920 to attempt to re-read the y-variable. - Regardless of the path taken to reach
step 920, if the read atstep 920 failed as determined bystep 922, then the arbitration is failed because some node reserved the replica member 58 1. Alternatively, if at step 92 4 the member's y-variable that was read changed from its value preserved in the local old_y variable, then a competing node appears to be ahead in the arbitration process, and the node 60 1 backs off as described below so that the other node can obtain the quorum. However, if the y-variable has not changed, it appears that no node is able to defend the replica member 58 1 and that the node 60 1 may be ahead in the arbitration, whereby atstep 924 the arbitration process continues to step 1000 ofFIG. 10 . - Note that it is possible for a plurality of nodes to successfully complete the challenge procedure of
FIG. 9 and reachstep 1000 ofFIG. 10 . In accordance with one aspect of the present invention, a mutual exclusion algorithm is executed to ensure that only one of the plurality of nodes succeeds in completing the arbitration process. In accordance with the principles of a fast mutual exclusion algorithm, atstep 1000 ofFIG. 10 , an attempt is made to write an identifier unique from other nodes to a second location, x, on the replica member 58 1. Note that as shown inFIG. 10 , for purposes of simplicity, any time a read or write operation fails, the arbitration is failed, and thus only successful operations will be described in detail herein. Then, steps 1002 and 1004 again test whether y's value on the replica member 58 1 still equals the old_y variable, since it may have just been changed by another node, e.g., node 60 3 wrote to y while the operation of writing the x value by the node 60 1 was taking place. If changed, at least one other node is apparently contending for ownership, and thus step 1004 backs off, i.e., fails the arbitration process. - If y is still unchanged at
step 1004,step 1006 generates a new unique myseq sequence identifier for the node 60 1 into the y location on the replica member 58 1, and if successful, continues to step 1008 where the value at the x location is read. If atstep 1010 the x location still maintains the my_id value (written at step 1000), then this node 60 1 has won the arbitration, reserves the disk atstep 1016 and returns with a success return code of “TRUE.” Alternatively, if atstep 1010, the x location no longer maintains the ID of the node 60 1, then apparently another node (e.g., 60 4) is also challenging for the right to obtain exclusive control. However, it is possible that the other node 60 4 has changed the x value but then backed off because the y-value was changed, (e.g., at its own steps 1002-1004), whereby the node 60 1 is still the leader. Thus, after a delay atstep 1012 to give the other node time to write to the y-location or back off, the y-value is read, and if the y value is changed atstep 1014, then the arbitration was lost. Note that a node which wins the arbitration writes the y-location immediately thereafter as described below with reference toFIG. 11 . - Conversely, if the y value is still equal to the unique sequence ID (myseq) of the node 60 1 at
step 1014, then this node 60 1 has won the arbitration, and returns with the “TRUE” success return code. Note that the mutual exclusion mechanism of steps 1000-1014 (run by each competing node) ordinarily ensures that only one node may ever reachstep 1016 to persist the reservation, because only the node having its ID in the y-location can enter this critical section, while the x-location is used to determine if any other nodes are competing for the y-location. However, there is a non-zero probability that more than one node will successfully complete the arbitration procedure, given arbitrary processing delays. This is because fast mutual exclusion depends on the delay atstep 1012 being long enough to guarantee that the participants that evaluated the condition atstep 1004 as true are able to write down their sequence number to the disk atstep 1006. However, if an unexpected delay occurs betweensteps step 1012, then more than one node could have successfully complete the arbitration procedure. This unlikely problem is made even less likely by the fact that a node places a SCSI reservation on a replica set member after successfully completing arbitration, as will be discussed later with reference toFIG. 11 . - Returning to
FIG. 8A ,step 806 evaluates the code returned for the current member from the single-member arbitration algorithm ofFIGS. 8 and 9 . If not successful, step 806 branches to step 808 to determine whether the failure to obtain control was caused by the member being owned by another node, or whether the member was inaccessible, e.g., crashed or not properly connected to the challenging node 60 1. If owned by another node, step 808 branches toFIG. 8B to determine whether the challenging node 60 1 already has a quorum, or should back off and relinquish any members controlled thereby as described below. If the failure occurred because the member was not accessible (as opposed to owned), step 808 branches to step 812 to repeat the process on the next member, as described below. - If at
step 806 it is determined that the challenging node 60 1 was successful in obtaining control of the replica member 58 1, step 806 branches to step 810. Atstep 810, the array tracking the node's control of this member is set to “TRUE,” the count used for determining a quorum is incremented, and the replica member 58 1 is set to be defended by the node 60 1 if the node 60 1 is able to achieve control over a quorum of the members. Defense of an owned member is described below with reference toFIG. 11 . Then, atstep 812, the current member is changed to the next member (if any) and the process returns to step 804 to again arbitrate for control of each remaining member of the total replica set of configured replica members. - Step 820 of
FIG. 8B is executed when the replica members have all been arbitrated (step 804 ofFIG. 8A ) or if an arbitrated replica member was owned by another node (step 808 ofFIG. 8A ) as described above. Step 820 tests whether the count of members owned by the challenging node 60 1 achieved a quorum. If so, step 820 returns to its calling location with a “TRUE” success code whereby the next step in forming a cluster will ultimately take place atstep 602 ofFIG. 6 , as described below. - If a quorum is not achieved, step 820 branches to step 822 to relinquish control of any replica members that the node 60 1 obtained ownership over, recompute the delay interval, and increment the retry (loop) counter. Step 824 then repeats the process after the delay interval at
step 826 by returning to step 802 ofFIG. 8A , until a maximum number of retries is reached. Typically the delay calculation instep 822 uses a well-known “exponential backoff” as follows:
BackoffTime=BackoffTime0*(EˆA n)*Rand ( )+BackoffTimeMin,
where BackoffTime0 is the maximum backoff time for the first try, E is a number greater than 1, typically 2 for convenience, n is the retry number (0 based), ˆ represents exponentiation (raised to the power), BackoffTimeMin is the smallest practical backoff time, and Rand ( ) is a function that returns a random number between 0 and 1. - If no quorum is achieved after retrying, the process ultimately returns to step 504 with a failure status.
Steps - It should be noted that
FIGS. 8A and 8B describe a probabilistic algorithm. In general, the ordering requirement, the restart of the process upon failure to control a member, and the random exponential backoff, when taken together, provide some non-zero probability that one of a plurality of independent (non-communicating) arbitrating nodes will successfully gain control of a quorum of the members in the set. The probability may be adjusted by tuning various parameters of the algorithm. Note that the use of exponential backoff techniques in arbitration algorithms is well known to those skilled in the art, e.g. it is the basis for CSMA/CD networks such as Ethernet. Moreover, note that the probabilistic nature of the overall algorithm is different than the probability that more than one node will successfully complete the arbitration procedure, given arbitrary processing delays, as described above. - Returning to step 602 of
FIG. 6 , when a quorum is achieved, an attempt is made to reconcile the replica members so that the correct cluster operational data may be determined. As described above, a requirement on any mechanism for maintaining the cluster operational data is that a change made to the data by a first instance of a cluster be available to a second instance of the cluster that is formed at a later time. In other words, no completed update may be lost. In order to meet these requirements for a set of replica members, changes pertaining to the update are applied to a quorum of the replica members, and an update is not deemed to be complete until this is successfully accomplished, thereby guaranteeing that at least one member of any quorum set has the latest data. In general, one way to accomplish this goal is to use a distributed consensus algorithm, such as one similar to the algorithm generally described in the reference entitled, “The Part-Time Parliament,” Leslie Lamport, ACM Transactions on Computer Systems 16, 2 (May 1998), 133-169. In order to reconcile the states of different members of a replica set, a quorum replica set algorithm, described below, is executed. In accordance with another aspect of the present invention, as part of the quorum replica set algorithm, a recovery process is initiated whenever a replica member becomes available and a majority of members are available. To determine the most updated member and thereby accomplish consistent reconciliation, an epoch number is stored on the log header of a log maintained on each replica member. The epoch on the log header is incremented during the recovery process and corresponds to the epoch that begins with that recovery process. In addition, every update is originally associated with an epoch number and a sequence number. These are stored on each replica member as part of the log record associated with this update. The epoch in the log records correspond to the epoch in which the update was made. - The failure of any read or write operation on a quorum replica set member during this recovery procedure is treated as a failure of the replica member, (although the operation may be optionally retried some number of times before declaring failure). A failed replica member is removed from the quorum replica set, as described below with reference to
FIG. 18 . The cluster may continue operating despite the failure of a member of a quorum replica set at any point, as long as the remaining set still constitutes a quorum. If the remaining set does not constitute a quorum, then the cluster must cease operating, at least with respect to allowing updates to the cluster operational data, as described below. If the quorum requirement is still met after a replica member failure, any update or reconciliation procedure that was in progress when the member failed continues forward unaltered, after the failed member has been removed from the quorum replica set. This procedure guarantees that all updates to the cluster operational data are sequentially consistent, that no committed update is ever lost, and that any cluster instance, which controls a quorum of the total replica set members, will have the most current cluster operational data. - If the reconciliation of the members at
step 602 is determined to be successful atstep 604, the process returns to step 504 ofFIG. 5 with a “TRUE” success status, otherwise it returns with a “FALSE” status. As described above, based on the status, step 504 either allows the cluster to operate or restarts the join/formation attempt up to some threshold number of times. - Step 700 of
FIG. 7A represents the performing of work by the cluster. In general, the work continues until some event occurs or a time of delta elapses, where delta is the arbitration time (e.g., three seconds) described above. Preferably, the node continues to perform work and runs a background process when an event/time interval is detected. Events may include a graceful shutdown, a failure of a replica member, and a failure of a node. Step 702 tests if a shutdown has been requested, whereby if so, step 702 returns to step 508 ofFIG. 5 with a TRUE shutdown status. Step 508 performs various cleanup tasks, and step 510 tests the shutdown status, ending operation of the node if TRUE. - If not a shutdown event, step 702 of
FIG. 7A branches to step 704 where the node makes a decision based on whether the node is the owner of the quorum of replica members. If so, step 704 branches to step 720 ofFIG. 7B , described below, while if not step 704 branches to step 706 where the quorum owner's communication with the node is evaluated. If the quorum-owning node is working, step 706 returns to step 700 to resume performing work for the cluster. Otherwise, step 706 branches to step 740 ofFIG. 7C , as described below. - Turning to
FIG. 7B , when a node e.g., 60 2 represents the cluster, atstep 720 the node tests whether an event corresponded to a failure of one or more of the replica members. If so,step 722 is executed to determine if the node 60 2 still has control of a quorum of replica members. If not, step 722 returns to step 508 ofFIG. 5 with a “FALSE” shutdown status whereby the cleanup operation will take place and the cluster join/formation process will be repeated for this node 60 2. However if the node 60 2 still has a quorum of members, step 722 branches to step 724 to defend ownership of each of the members, as described below. Note that the defense of the members (FIG. 11 ) is essentially performed on each member in parallel. - As shown at
step 1100 ofFIG. 11 , to defend each of the owned replica members, the node 60 2 first sets a loop counter for a number of write attempts to zero, and then attempts to exclusively reserve that member, e.g., via the SCSI reserve command. If unsuccessful, another node has won control of this disk, whereby the node 60 2 re-evaluates atstep 726 ofFIG. 7B whether it still possesses a quorum. If the node has lost the quorum, the node 60 2 will ultimately return to step 508 ofFIG. 5 and repeat the join/formation process. - If successful in reserving the disk,
step 1104 is next executed where a new myseq value is generated for this node 60 2 and an attempt is made to write to write the y-variable used in the arbitration process, as described above. The y-variable is essentially rewritten to cause other nodes that are monitoring the y-value after breaking the previous reservation to back off, as also described above. If the write succeeds, the replica member was successfully defended, and the process returns to step 726 ofFIG. 7B with a “TRUE” success status. If the write failed,steps - Returning to step 726 after attempting to defend the members, if the node 60 2 no longer has a quorum, the node returns to step 508 of
FIG. 5 to cleanup and then repeat the join/formation process. Conversely, if the node still possesses a quorum of the members,step 728 is next executed to test whether the node 60 2 that represents the cluster owns all the members of the total replica set 106 of configured members. If so, step 728 returns to step 700 ofFIG. 7A . However if not all the members are owned, for reliability and robustness, the node representing the cluster attempts to obtain control of as many of the operational replica members as it can. Thus, atstep 730, the node attempts to gain control of any member, M, for which OwnedMember(M)=FALSE, using the single member arbitration algorithm ofFIGS. 9 and 10 described above. If there are multiple members that are not owned, the node may attempt to gain control of them in any order, or in parallel. -
FIG. 7C represents the steps taken by a node (e.g., 60 1) that is not in control of the quorum replica set (step 704 ofFIG. 7A ) and that is no longer communicating (step 706 ofFIG. 7A ) with the node that was in control of the quorum replica set. First,FIG. 7C calls the process (beginning atFIG. 8A ) that arbitrates for control of the replica members of the total replica set. If a quorum is not achieved as ultimately evaluated atstep 740,step 742 is executed to determine if the node 60 1 is now communicating with the quorum owner. Note that ownership may have changed. If connected atstep 742, the node 60 1 returns toFIG. 7A to perform work for the cluster, otherwise the node returns to step 508 ofFIG. 5 to cleanup and restart the joining, formation process as described above. - Alternatively, if at
step 740 the node successfully acquired control over a quorum of replica members,step 744 is executed to reconcile the quorum members and form the cluster as described above. If successful in reconciling the members, the node 60 1 returns toFIG. 7A to perform work for the cluster it now represents, including executing the steps ofFIGS. 13-24 as appropriate, otherwise the node returns to step 508 ofFIG. 5 to cleanup and restart the joining, formation process as described herein. - In alternative implementations, not all of the cluster operational data need be maintained in the replica members 58 1-58 3, only the data needed to get the cluster up and running, as described in the aforementioned copending U.S. patent application entitled “Data Distribution in a Server Cluster.” In one such alternative implementation, the replica members maintain this “core-boot” data, and also maintain information regarding the state of the other cluster operational data, (e.g., configuration information about the applications installed on the cluster and failover policies). The state information ensures the integrity of the other cluster operational data, while the other storage device or devices (e.g., a mirror set of storage elements) that store this data provide a relatively high-performance and/or lower cost storage for this additional cluster configuration information, with high reliability. In any event, as used herein, the replica members 58 1-58 3 maintain at least enough information to get a cluster up and running, but may store additional information as desired.
- Note that a quorum need not be a simple majority, but may, for example, be some other ratio of operational members to the total number, such as a supermajority (e.g., three of four or four of five). However, a primary benefit of the present invention is to provide availability with the minimum number of components, and such a supermajority requirement would tend to reduce availability.
- Instead, cluster availability may be increased by requiring only a simple majority while using a larger number of devices. For example, three replica members may be configured for ordinary reliability, in which two disks will have to fail to render the cluster unavailable. However, the more that reliability is desired, the more replica members may be used, (at a cost tradeoff), e.g., three of five failures is less likely than two of three, and so on. Note that SCSI limitations as to the number of replica members and their physical separation need not apply, as described in U.S. patent application Ser. No. 09/260,194 entitled “Method and System for Remote Access of Network Devices,” assigned to the same assignee as the present invention.
- The Quorum Replica Set Algorithm
- While having a set of multiple replica members increases cluster availability and reliability over a single quorum device, having a replica set requires providing consistency across the members of the replica set. This consistency needs to be maintained even though individual replica members can fail and recover at various times.
- In accordance with another aspect of the present invention, to keep a replica set consistent in view of replica failures and recoveries, (and also following a temporal partition), a quorum replica set (QRS) algorithm is provided that among other things, performs a recovery process whenever a change to a replica set occurs, (that is, any time a formerly unavailable replica member becomes available). The QRS algorithm also prevents updates when less than a quorum of replica members is available. To this end, as part of the QRS algorithm, any time a write occurs to a replica member, (described below with respect to
FIGS. 20-22 ), the success of that write determines whether the replica member is available or has failed. If failed, the remaining available set is checked for a majority, and no updates are allowed unless there is a majority. - The QRS algorithm is capable of being run on any node that is capable of representing the cluster via ownership of the quorum replica set 57. The QRS algorithm may be run during startup as replica members are detected, e.g., to bring those members online, and is also run during normal execution, such as by the node that possesses the quorum replica set 57, to ensure that replica members that come online or go offline are properly dealt with, and to ensure that data updates only occur when a quorum of configured members exists. For purposes of simplicity, the QRS algorithm will be primarily described with respect to its operation after a cluster has already been formed.
- The QRS algorithm includes three properties. A first property is that configuration information updates that are applied to a majority of the members of a replica set will never be undone or lost despite the subsequent failure and recovery of replica set members and/or nodes executing the QRS algorithms. A second property is that an update that was recorded to some of the replica members but not committed in a previous recovery, or an update that was made without the knowledge of a previously committed update in a later epoch, will not get committed during recovery. A third property is that an update is reported to have succeeded if and only if the update was applied to a quorum of the replica members. Thus, if an update was in progress when a failure occurred, but had not yet been applied to a quorum of the replica members, then its fate cannot be known until recovery is complete. Such an update may be either committed or discarded during the recovery procedure. When an update is committed to at least the quorum, the update is reported to the cluster as having been successfully committed. Such reports (commit notifications) are generated in the same order in which the updates occur.
- In order to ensure replica consistency, the QRS algorithm uses a log (a standard database technique) that logs the updates in records on each replica member, including three variables associated with the logged information. For example, in the three-member configured replica set generally represented in
FIGS. 12A-12D , each replica member (e.g., 0, 1 and 2) includes a respective log 120 0-120 2. In each log, a first variable is a replica epoch number, 122 0-122 2, respectively, which is a number stored in a header 124 0-124 2 of the log on each replica. The replica epoch number, also referred to herein as a current replica epoch, is associated with a recovery session as described below, and always moves in one direction (e.g., increases by at least one during the recovery process). - A second variable used by the QRS algorithm is an update epoch number. The update epoch number is stored with each logged record to associate that update record with the current replica epoch value at the time the update record was logged. In
FIGS. 12A-12D , the log sequence number is represented by the box in each record (e.g., Rec1.0) under the italicized letter “E.” - A third variable is a log sequence number, that tracks the relative sequence of each logged update record with respect to other logged update records. In
FIGS. 12A-12D , the record epoch is represented by the box in each record (e.g., Rec1.0) under the italicized letter “S.” After a successful recovery, the sequence numbers are guaranteed to be the same for the logs on every replica member that is part of the currently available set of replica members. In particular, it must not be possible for two different update records that were applied by two different cluster instances to have the same update epoch.sequence number. Note that in addition to the update epoch number and log sequence number, each record also will typically (e.g., except for certain NULL data instances) contain the update data that describes the update. - The QRS algorithm will be described herein with reference to the general flow diagrams of
FIGS. 13-24 and the above-described replica epoch, update epoch and sequence number variables. One part of the QRS algorithm is directed to handling replicas that are configured for cluster operation, but were unavailable for some reason, and then become available for operation. For example, having another replica member become available may cause a quorum of replica members to be achieved, (where there previously was less than a majority of members available), whereby updates then become possible. Another part of the QRS algorithm operates during a requested data update. Only when a majority of replica members have committed an update is an update reported as having been successful. Alternatively, reads and updates may be prevented from even being attempted if the QRS algorithm has detected that a majority of replica members are not available. In addition, an update attempt may fail because a replica member has failed, in which event a majority may no longer be available and further data updates need to be prevented. -
FIG. 13 logically represents these related parts of the QRS algorithm. For example, as shown inFIG. 13 , viastep 1300, when a new replica member becomes available to the quorum replica set, a Replica Online process (beginning atFIG. 14 ) is executed. If instead a new replica member becomes unavailable to the quorum replica set, viastep 1302, a Replica Offline process (FIG. 22 ) is executed. Alternatively, when a data read is being requested,step 1304 calls a read process to handle it, while when a data update is being requested,step 1306 calls an update process to handle the update request. Note that for simplicity,FIG. 13 shows a process looping forever to handle a replica member becoming available/unavailable or a data read/update. However, as can be readily appreciated, instead of executing such a loop, such detections are typically event driven in response to an appropriate event. -
FIG. 14 represents the QRS algorithm when a new replica member becomes available to the quorum replica set. As can be appreciated, this can be detected in many ways, such as by occasionally polling for a replica member, via plug-and-play type detection or via similar event notification. Note thatFIG. 14 handles typical situations wherein the newly available replica member is already configured for operation in the cluster, (i.e., is already known to the cluster and thus one of the total possible), but was previously unavailable to the cluster nodes. For example, unavailability can happen if a replica member is disconnected for some reason, including inadvertently (e.g., accidentally unplugged) or intentionally (e.g., for maintenance) reasons, wherebyFIG. 14 operates when such a replica member is reconnected. -
FIG. 14 represents a replica becoming available, and begins atstep 1400 wherein an update lock is acquired. The update lock prevents possibly conflicting processes that are running at the same time from changing global variables, e.g., the process ofFIG. 14 may have to wait to acquire the lock if the update process ofFIG. 20 is running (and thus possesses the update lock). Note that the update lock provides a simplified scheme for serializing updates and reads, however other, more sophisticated schemes may provide better performance, e.g., by enabling concurrent reads and/or concurrent writes to different data elements, and as such, these alternative schemes may be employed. -
Step 1402 prevents updates from occurring during operation of the replica online process, such as by setting an update variable to FALSE. For example, as will be described below, the update process ofFIG. 20 exits if updates are not allowed. Note that the replica online process ofFIG. 14 will re-enable updates (at step 1412) if certain conditions, described below, are met. -
Step 1404 increments a count of the number of available replicas, to reflect the detection of the newly-available replica that triggered operation ofFIG. 14 .Step 1406 adds an identifier of this replica to a set that maintains which replicas are currently available. Then,step 1408 represents the test for whether a quorum (e.g., majority) has been achieved based on the actual available count versus the number required for a majority, (which is known to the cluster nodes). If there is no majority, step 1408 branches ahead to step 1414 to release the update lock, after which the replica online process ends. Note that via the above-describedstep 1402, updates are precluded in this situation. - If however a majority of replicas are available (step 1408), then a recovery process is started, as generally represented via
FIGS. 15A-15B . The recovery part of the QRS algorithm is thus executed when a majority of replica members are available from those that are configured for cluster operation. In general, the recovery process operates to make the replica members consistent with one another, so that possession by a node of any majority of replica members ensures that the latest changes are known to the cluster in any given quorum replica set. Note that although not shown inFIGS. 15A and 15B for purposes of simplicity, if an operation fails at an appropriate place, for example, a write to log, opening of a log, propagating a record to a log, or the like, recovery is aborted (viaFIG. 18 , described below) and a FALSE status is returned toFIG. 14 as the recovery status to indicate the lack of success. Further note that not shown in each possible instance, this inherent abort-on-failure situation (FIG. 18 ) applies when appropriate with respect toFIGS. 16 and 17 , which are part of the recovery process. - Following recovery,
step 1410 ofFIG. 14 will test for success, and if success status is FALSE, regardless of where in the recovery process it was generated,step 1410 will prevent updates, essentially by bypassing step 1412 (which if executed would re-enable updates) and instead branching ahead to step 1414 to release the update lock.Step 1412 is thus only executed to re-enable updates if the recovery process was successful. Note, however, that it is alternatively feasible for a cluster to allow updates as long as a majority of replicas is still available, e.g., there is no reason to halt updates when a majority exists before a new replica member is detected but that new replica fails during recovery, as long as a majority still exists afterward. For simplicity, only successful operations will be described hereinafter in the recovery process, except as otherwise noted. -
FIG. 15A begins by first calling an initialize process ofFIG. 16 that initializes the above-described log of each available replica. More particularly,FIG. 16 tests whether a log-opened variable equals TRUE, indicating the replica is already initialized. If not initialized (the variable is set FALSE inFIG. 18 orFIG. 22 when a replica goes offline, as described below), indicating that the replica member is offline, initialization is attempted to make the replica member available. - If the replica is not already initialized at
step 1600, then the log file is opened atstep 1602. If the log is not new, then it is mounted viasteps step 1606 fails,step 1607 aborts the recovery and takes this replica offline as described below with reference toFIG. 18 . Further, note that if the read atstep 1606 is successful, the log of each quorum replica set member is replayed viastep 1608 during initialization, to ensure that the replica member's data is self-consistent. Any partially written records are discarded (undone). Following step 1610 (which also can have a read failure), the process then advances to step 1630 (entry Point 2) ofFIG. 16B , which sets the “log-opened” variable to TRUE for this log.Step 1632 sets a recovery log header variable maintained in node local storage for this particular replica equal to the log header variable. - If the log was just created, then it is initialized via
steps FIG. 16B . - At
step 1620 ofFIG. 16B , a starter record is prepared for the log on the replica, with a record header epoch equal to the local log header epoch variable (initialized to zero), the local record header sequence equal to the log header sequence variable (initialized to zero), and NULL record data.Step 1622 attempts to write this record to the log. If the write fails,step 1624 calls the abort recovery process ofFIG. 18 , described below. If successful, the local log header sequence variable is incremented (for the next update). As described above, once the log for this replica member is initialized, the “log-opened” variable is set to TRUE for this log atstep 1630, and step 1632 sets a recovery log header variable maintained in node local storage for this particular replica equal to the log header variable. The process then returns to step 1502 ofFIG. 15A . -
Step 1502 ofFIG. 15A is thus executed following the various log initialization operations ofFIGS. 16A-16B . Based upon the epoch numbers recorded in the header of each of the available replicas, a maximum epoch number is determined atstep 1502. A current replica epoch is established by adding one to the maximum atstep 1504, and the current replica epoch is written to the log headers on all the replicas in the availability set such that they are updated to the current replica epoch. Note that although not specifically shown, a write failure results in the abort recovery process ofFIG. 18 being executed. -
Step 1506 represents the reading of the last two valid records (one record if only one record exists, e.g., the starter record) from the log of each available replica. Again, although not specifically shown, a read failure results in the abort recovery process ofFIG. 18 being executed. - At
step 1508, from among the replicas, the replica (or replicas) having a last record with the highest update epoch number is chosen as a candidate for leader. If atstep 1510 only one replica has the highest epoch number in its last record, there is only one candidate for the leader replica, and it is selected as the leader atstep 1512. In the event of a tie in record epochs, atstep 1514, a leader is selected from the leader candidates based on the highest log sequence number in its last record. In other words, the leader is a replica member having in its last record an epoch.sequence number greater than or equal the maximum epoch.sequence number of the last record on the available replicas. Note that if two or more candidates replicas have the same sequence number, any one of those can serve as the leader replica since each have the same last record, however another tiebreaker may be used based on some other criteria if desired. For example, if an epoch.sequence tie exists, the replica with the log having the lowest log identifier becomes the leader replica. Further, if all available replicas each the same epoch and sequence number for their respective last record, then no propagation of records (FIG. 15B ) is needed, and these steps can be avoided. In the present example, for purposes of explanation, it is assumed that this is not the situation at this time. - Once a leader replica is selected, the recovery process continues to
FIG. 15B , to propagate any needed records from the leader to other replicas. Atstep 1520, the last record in the replica log of the leader replica is retagged with the current replica epoch. -
Step 1522 selects a replica that is not the leader for updating. Based on the last two records therein (previously read via step 1502), the records that are needed to update that non-leader replica relative to the leader replica are determined atstep 1524. These records, referred to as the set of records to update, or recordset, will be propagated to the selected non-leader replica via the process ofFIG. 17 . In other words, the necessary records from the leader replica (greater than or equal to the Epoch. Sequence of the second last record on a non-leader replica) are propagated from the leader replica to the other replicas. - During the propagation, the last two records on the non-leader replicas need to be examined with respect to the records being propagated by the leader replica, because the last record may correspond to an update that was made to this replica but that was not committed to a majority of the replicas and now conflicts with an update committed in a later epoch, while the second last record may have been retagged in a previous unsuccessful recovery session. This part of the QRS algorithm, shown in
FIG. 17 , essentially determines whether to discard or retag the records in the selected non-leader replica by comparing the first two records in the recordset sent by the leader replica against the last two records on that selected non-leader replica. Thereafter, any remaining records in the recordset sent by the leader replica are applied to the selected non-leader replica to make it consistent. - More particularly,
step 1700 ofFIG. 17 first tests whether there is any second to last record on the selected non-leader replica. If not, step 1700 branches to step 1702 where the last record is evaluated (at least the starter record will exist) against the first record in the set of propagated records. If the records are not the same, atstep 1704 the last record in the non-leader is replaced (atomically) with the first record in the propagated recordset from the leader. Atstep 1706, the remaining records in the recordset propagated from the leader replica are applied, whereby this selected non-leader replica is consistent. Although not specifically shown, as mentioned above, if any read or write failures occur, the recovery process is aborted viaFIG. 18 , described below. - If instead step 1700 determines that the second to last record on the selected non-leader replica exists, then step 1700 braches to step 1710 where the second to last record in the selected non-leader replica is evaluated against the first record of the leader's propagated recordset. If the same, step 1710 branches to step 1712 to evaluate the last record of the selected non-leader replica against the second record of the leader's propagated recordset. If these are not the same, then the last record of the non-leader replica is atomically replaced by the second record of the leader's propagated recordset at
step 1714. Any remaining propagated records are then applied viastep 1716. If instead step 1712 determines that the epoch and sequence for the records match, step 1712 branches to step 1718 wherein any remaining propagated records are then applied. - Returning to step 1710, if the second to last record in the selected non-leader replica is not the same as the first record of the leader's propagated recordset, step 1710 branches to step 1720 where the last record of the non-leader replica is discarded.
Step 1722 then replaces the second to last record of the non-leader replica with the first record in the leader's propagated recordset, and then step 1724 applies any remaining propagated records from the leader's propagated recordset to the selected non-leader replica. - The process of
FIG. 17 ultimately returns to step 1526 ofFIG. 15B , such as to determine whether another non-leader replica needs to be updated. If so,step 1528 selects that non-leader replica as the selected non-leader replica, and the process ofFIG. 17 is similarly executed therefor. Note that steps 1522 to 1528 are generally represented as showing the propagation of the leader's records to each of the non-leader replicas to one non-leader replica at a time. However, as can be readily appreciated, some or all of these propagation-related steps may be performed to multiple non-leader replicas in parallel. - When the non-leader replicas have been made consistent with the leader replica,
step 1530 is performed to report (generate the commit notifications for) the successful committing of the last record transmitted from the leader replica. For efficiency, such commit notifications only have to be generated for records propagated since the last recovery. - At this time, recovery is complete, and step 1532 returns to step 1410 of
FIG. 14 where the success of the recovery process is evaluated. If successful,step 1412 is executed to allow updates, and the replica online (including recovery) process completes by releasing the update lock (step 1414). - As mentioned above, if any read or write failure to a replica occurs during the recovery process, the abort recovery process of
FIG. 18 is called with the identity of that replica. This function is called with the replica id of the bad replica if the recovery process fails. Note that the update lock is held when this function is invoked. Atstep 1800, a count of the number of available replicas is decremented, andstep 1802 removes the identifier of this replica from the set that tracks which replicas are currently available, to reflect that this replica is no longer available.Step 1804 forces the log to be initialized again when the replica subsequently comes online by setting the “log opened” variable to FALSE for this replica. As described above, this variable is evaluated atstep 1600 ofFIG. 16 , prior to initialization. A variable indicative of success (evaluated atstep 1410 ofFIG. 14 ) may also be set atstep 1806 to indicate that recovery failed.Step 1808 then generates an event that will ordinarily cause other processes in the system to try and get this replica member online again, check for its integrity, and so forth.Step 1810 generates another event, a recovery event, which will restart recovery if a majority of replica members is present. Generating this recovery event guarantees that if this replica does not recover or come online, the recovery process will be retried again as long as majority of replicas exists. Note that it is alternatively feasible to haveFIG. 18 test for whether a majority of replica members is consistent, and if so, to not consider the recovery to have completely failed (which requires a restart of the recovery process. -
FIG. 19 represents a replica record read operation (of one replica member) consistent with the QRS algorithm. Note that this is a replica read in ordinary operation i.e., not during the replica online/recovery process. If one-copy serializability is desired (a property which guarantees that concurrent execution of transactions on replicated data is equivalent to a serial execution on non-replicated data), such reads are not allowed until a majority of replicas is available. - In
FIG. 19 , read operations acquire the update lock atstep 1900, and prevent read operations at any time that updates are not allowed viastep 1902. An attempt to read while updates are not allowed is considered an error viastep 1904. If the read attempt is allowed,step 1906 attempts to read the requested recordset and returns a status value equal to the success or failure of the read attempt. Note that if a read failure occurred, this replica is taken offline as described below with respect toFIG. 22 , and this read can be retried on another member if a quorum still exists. Before returning to the process that requested the read, the read operation releases the update lock atstep 1908.FIG. 20 represents a replica update (write) request handled in conjunction with the QRS algorithm and its properties. Atstep 2000, a counter that tracks the number of successful writes is initialized to zero, and atstep 2002, the update lock is acquired as described above.Step 2004 then tests whether updates are currently allowed. As described above, updates are not allowed unless a quorum of consistent replica members is available. If not allowed, step 2004 branches to step 2006 where the update lock is released, and an error is returned viastep 2008. - If updates are allowed,
step 2004 instead branches to step 2010 wherein an attempt to make the update is made, e.g., a data write attempt, to each available replica member.FIG. 21 represents the actions taken on each replica member in the write update attempt. Note that the write attempts may be made in parallel. - At
step 2100 ofFIG. 21 , the log header variable of the replica member is set to equal the recovery log header variable for this replica (consistent withstep 1632 described above), and the sequence number variable is increased atstep 2102. To build the update record, the epoch number for the record is set to equal the epoch number stored in the local node's log header for this recovery epoch, as described above. Similarly, the sequence number for the record is set to equal the just-incremented sequence number stored in the log header. Lastly, the record's data field is set to include the data that is to be written atstep 2108. Note that any checksums or the like can be calculated and added to the record at this time. When ready, an attempt to write the record is made atstep 2110. -
Step 2112 evaluates whether the write was successful. Note that although not shown, any writes to the replica member are not to be cached but instead written through to the disk. If the record is successfully written (and flushed) to the disk, a TRUE status is returned (to step 2012 ofFIG. 20 ) as the status of the operation. If either the write (or any flush operation) was not successful, then FALSE is returned (to step 2012 ofFIG. 20 ) as the status. -
Steps 2012 through 2020 ofFIG. 20 work with the returned write status from each replica, and thus are executed for each of the replicas.Step 2012 evaluates the write status for a given replica. If not successful, updates (to any replica) are prevented viastep 2014, and the particular replica on which the write failed is declared offline atstep 2016, e.g., by generating an offline event or the like that will cause the offline process ofFIG. 22 to be called. The process for handling an offline replica is described below with respect toFIG. 22 , however at this time it should be pointed out that among other things, when a replica goes offline, the offline handling process re-enables updates if a majority of replicas are still available. For a write that was successful, the write counter is incremented atstep 2020. - When a write status has been returned from
FIG. 21 for each replica,step 2022 compares the number of successful writes in the counter against the majority number that is required for a quorum. If a majority was not successfully written, then a FALSE status is returned as the update status viastep 2024 to the process that requested the update. Note that whenstep 2024 is executed, updates are not allowed (viastep 2014. The update lock is released viastep 2032. - If instead at step 2022 a majority was successfully written,
step 2026 is executed which reports that this record was successfully committed.Step 2028 re-enables further updates since a majority of writes were known to be successful. A TRUE is returned viastep 2030, and the update lock is released viastep 2032. -
FIG. 22 represents the offline process executed when a replica has become unavailable. Note that the described offline process is executed for each unavailable replica rather than handling multiple unavailable replicas at once, although such a process is feasible. Further, note that as described above, a replica can be declared unavailable because of a failed write viastep 2016, or a failed replica member can be detected in some other manner (e.g., via a failed read, as described above). In any event, the offline process begins atstep 2200 wherein the update lock is acquired to prevent possibly conflicting processes running at the same time from changing global variables. -
Step 2202 decrements a count of the number of available replicas, andstep 2204 removes the identifier of this replica from the set that tracks which replicas are currently available, to reflect that this replica is no longer available. Note that steps 2202 and 2204 are essentially counter to thesteps Step 2206 forces the log variable in the recovery structure to be initialized again when the replica subsequently comes online by setting a variable or the like for this replica. As described above, this variable is evaluated atstep 1600 ofFIG. 16 , prior to initialization. -
Sep 2208 represents the test for whether a quorum (e.g., majority) still exist based on the count that remains versus the number required for a majority. If there is not a majority, step 2208 branches to step 2210 to disable updates. If there is a majority,step 2208 instead branches to step 2212 to allow updates. After either step, the offline process continues to step 2214 to release the update lock, after which the replica offline process ends. - Returning to
FIGS. 12A-12D , an example will now be provided of the general operation of the QRS algorithm as described above. InFIG. 12A , two replica members (0, 1) are available from a configured set of three replica members (0, 1, 2), wherein the logs 120 0-120 2, replica epochs 122 0-122 2 and headers 124 0-124 2 of each have the replica member number as a subscript. Note that inFIGS. 12A-12D , the large diagonally crossed lines indicate the unavailability of whichever replica member is crossed-out. - In
FIG. 12A , the current replica epochs 122 0 and 122 1 in respective headers 124 0 and 124 1 are both at 1. As also shown inFIG. 12A , update 1.0 has been logged in bothreplica logs -
FIG. 12B represents the next replica epoch, whereinreplica members FIG. 12B ,replica members FIG. 12A ) in any record was 1. In addition, as described above,replica 1 is chosen as the leader replica, since prior to recovery,replica 1 had a record therein with a record epoch equal to 1, whereasreplica 2 had only the starter record. As also shown in the changes fromFIG. 12A toFIG. 12B , during recovery, the record of replica 1 (1.0 inFIG. 12A ) is retagged to 2.0, and this record is propagated toreplica member 2. The starter record is replaced as described above with respect toFIG. 17 , after which the recovery process considers the update successful. - As also represented in
FIG. 12B , while later operating,replica members respective logs FIGS. 20 and 21 . Still later an update record 2.2 is written toreplica member 1 but not toreplica member 2, as in this example, the node owning and controlling the replica member dies or shuts down unexpectedly. Again, since this update was not recorded to a majority, the change corresponding to this update record is not acknowledged as having been committed. - Sometime later, as generally represented in
FIG. 12C ,replica member 0 comes online, whereby the replica majority is achieved viamembers replica FIG. 12B ). In addition, as described above,replica 2 becomes the leader, since it had the record therein with a record epoch equal to 2, whereas replica O's largest record epoch number was a 1. As also shown in the changes fromFIG. 12B toFIG. 12C , during recovery, the record 1.1 ofreplica 0 is discarded, because this last record was determined (viaFIG. 17 , described above) to have not been committed to the quorum replica set prior to propagated (retagged) record 3.1 ofreplica 2 having been committed. Replica record 3.1 thus overwrites this record in thelog 120 0, and the recovery process reports the update as being successfully committed. In the example,replica member 0 then goes offline without any other updates having occurred. - In the last part of the example, generally represented in
FIG. 12D ,replica member 1 comes online, whereby the replica quorum is now achieved viamembers replica replica 2 is chosen as the leader, since it had the record therein with a record epoch equal to 3, whereasreplica 1's largest record epoch number was a 2. As also shown inFIG. 12D , during recovery, the second-to-last record 2.1 ofreplica 1 is kept and retagged to 4.1, while the last record, 2.2, is discarded as being not having been committed prior to a subsequent record having been committed. As can be readily appreciated, regardless of which replica fails and/or when it fails, the QRS algorithm ensures that no record which is successfully committed is ever lost. At the same time, the QRS algorithm ensures that records that were not committed to a majority are not kept if a subsequent update was committed first. Lastly, reports of successfully committed updates (commit notifications) are generated in the same order in which the updates occur. - The above description and accompanying examples are directed to handling replica members becoming available or unavailable when the total configured replica set is constant. However, the QRS algorithm can also handle the situation wherein new, previously unknown replica members are added to the total configured replica set, or when previously configured members are removed from the total configured replica set.
- As can be readily appreciated, changing the number of replica members in the total configured replica set changes the majority requirement, which if done incorrectly could cause a significant problem. When adding replica members, care must be taken to ensure that in the event of a cluster or replica member failure during the addition process, a subsequent majority cannot be allowed without at least one member present from the prior epoch. For example, it cannot be possible to change from a two of three requirement to a three of five requirement prior to making the new replicas consistent, otherwise data could be lost. By way of example, if a first quorum set is operating with only replica members A and B available out of a total configured replica set of A, B and C, replica member C is inconsistent. If new replica members D and E are then added, and the majority requirement becomes three of five, forming a new cluster with only replica members C, D and E cannot be allowed, unless at least one of C, D and E are first updated to include A and B's data.
- Also, when adding a replica member and thus changing the majority requirement, the change needs to be done such that a majority can later be achieved regardless of failures. For example, if only two replicas (A, B) are available out of three replicas (A, B and C) configured, and the number of the total configured replica set is increased to four by the addition of replica member D, then three replicas will be needed for a majority. If however, after increasing the majority requirement the cluster and the replica D fail while making D consistent, then only A and B may be available, which will not achieve a quorum.
-
FIG. 23 describes the addition of a new replica member (or members) to the total configured replica set in a manner that handles failures. Before a new replica is added, however, atstep 2300 its local header information (metadata) is written to be worse than any real replica so that it will never be selected as a leader, e.g., its replica and update epochs are set to negative values (e.g., to −1, −1). Note that to speed up the recovery process, it is feasible to lazily copy data to the new replica before it is actually added to the quorum replica set, however its replica and update epoch metadata will remain at −1, −1 until changed in actual recovery. - At
step 2302, the update lock is acquired, and the update process (ofFIG. 20 , described above) is called to make a change to the quorum configuration information maintained in the replica set, namely to record that a new replica is being added to the total configured replica set. Atstep 2304, further updates are prevented, until re-enabled as described above. If the update was successful, as evaluated atstep 2306, the new replica is recognized (is brought online) atstep 2308, (similar to the online process described above with respect toFIG. 14 ). If the update failed, a recovery event is issued, which among other this will re-enable updates if a majority of configured replicas is available, as described above. - If the update is successful (step 2306) and the replica is now online (step 2308), then the above-described recovery process (of
FIGS. 15A-15B ) is started to make the new replica member consistent with the set. If recovery is successful atstep 2310, then the status is set to TRUE atstep 2312, the update lock is released atstep 2316, and the status returned (step 2318). - If either the update failed (step 2306) or recovery failed (step 2310), the status is set to FALSE at
step 2314. The update lock is then released atstep 2316, and the status returned (step 2318). Note that if the update was not successful, this change might get committed to the current majority during the next recovery. If so, recovery can keep track of this special update, and reenter recovery. Further, note that failure during the writing of the epoch metadata (−1, −1) will leave the new replica in an uninitialized state, so it will never be visible to the cluster. Still further, note that failure during the update may leave the system in a state where some of replicas are aware of the new member or members, while others are not. If during subsequent recovery a replica member that is aware of a new member is operating in the quorum set, the new member information will get propagated, and the new member will be included and made consistent. If no member of a new quorum replica set is aware of the new member, an extra member will be visible but will be ignored by the cluster since it will not be part of the total configured replica set. - When removing (decommissioning) a replica member from the total configured replica set, care similar to that described above is taken to ensure that the problems above are not encountered in the event of failures, namely that data is not lost, and that a majority can still be achieved after removal.
FIG. 24 describes the removal of a replica member (or members) from the total configured replica set in a manner that handles failures. Atstep 2400, the update lock is acquired, and atstep 2402 the replica is tested for whether it is part of the available set, i.e., is online. If so, step 2402 branches to step 2404 which takes the replica offline. - Next, the update process (of
FIG. 20 , described above) is called to make a change to the quorum configuration information maintained in the replica set, that is, to record that a replica is being removed from the total configured replica set. One reason that the update may fail is that bringing the replica member offline causes the majority to be lost. However, if the update was successful, the recovery process will be correct, since the change will be on the old majority of replicas and consequently will be on a new majority of replicas. - At
step 2406, further updates are prevented, until re-enabled as described above. If the update was successful, as evaluated atstep 2408, then the above-described recovery process (ofFIGS. 15A-15B ) is started to ensure that the remaining replica members are consistent in the set. If recovery is successful atstep 2410, then the status is set to TRUE atstep 2412, the update lock is released atstep 2416, and the status returned (step 2418). - If either the update failed (step 2408) or recovery failed (step 2410), the status is set to FALSE at
step 2414. The update lock is then released atstep 2416, and the status returned (step 2418). Note that if the update was not successful, this change might get committed to the current majority during the next recovery. If so, recovery can keep track of this special update, and can reenter recovery. - As can be seen from the foregoing detailed description, there is provided a method and system for increasing the availability of a server cluster while reducing its cost. By requiring a server node to own a quorum of replica members in order to form or continue a cluster, and maintaining the consistency of the replica members, integrity of the cluster data is ensured.
- While the invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and has been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.
Claims (24)
1. A computer-implemented method, comprising:
maintaining cluster operational data on a replica set comprising a plurality of replica members that are each independent of any node of a server cluster;
representing the cluster at a node if the number of replica members controlled by the node comprises at least a majority of the total number of replica members configured to operate in the cluster; and
determining which of the replica members of the replica set has operational data that is most updated, and replicating at least some of that operational data to the other replica members of the replica set.
2. The method of claim 1 wherein determining which of the replica members of the replica set has the most updated operational data includes, maintaining an epoch number in association with each replica member.
3. The method of claim 2 wherein a value of each epoch number indicates a relative state of the cluster operational data on its respective replica member, and wherein determining which of the replica members of the replica set has operational data that is most updated includes determining which of the epoch numbers from each member is largest.
4. The method of claim 3 wherein at least two members have epoch numbers that equal the largest epoch number, and wherein determining which of the replica members of the replica set has the most updated operational data includes, maintaining a sequence number in association with the cluster operational data, and determining a largest sequence number from the replica members that have epoch numbers that equal the largest.
5. The method of claim 1 further comprising, evaluating a last record logged on a replica member to which data is being replicated, against at least one record of the replicated data, to determine whether to discard the last record.
6. The method of claim 5 further comprising, evaluating a second-to-last record logged on the replica member to which data is being replicated, against at least one record of the replicated data, to determine whether to discard the second-to-last record.
7. The method of claim 1 further comprising, detecting availability of a new replica member that is configured to operate in the cluster, and reconciling the cluster operational data of the new replica member.
8. The method of claim 1 further comprising, detecting the unavailability of a replica member that was operational, determining whether the majority of replica members still exists, and if not, halting updates to the cluster configuration data and executing a recovery process to attempt to obtain control of a majority of replica members.
9. The method of claim 1 wherein maintaining the cluster operational data includes storing information indicative of a total number of replica members configured in the cluster, and/or storing state of at least one other storage device of the cluster.
10. The method of claim 1 wherein the node controls the majority of replica members by arbitrating for exclusive ownership of each member.
11. The method of claim 10 wherein arbitrating for exclusive ownership includes executing a mutual exclusion algorithm.
12. The method of claim 1 wherein the node controls the majority of replica members by arbitrating for exclusive ownership of each member, including, issuing a reset command.
13. A computer-implemented method, comprising:
storing cluster operational data on a replica set of at least one replica member, each replica member being independent from any node;
at a first node, arbitrating with at least two other nodes for control of the replica set, the arbitration being performed for each replica member and comprising, attempting to obtain a right to exclusively reserve that replica member, and if the attempt is successful, exclusively reserving that replica member; and
representing a server cluster at the first node if the replica set is controlled by the first node and has consistent cluster operational data with respect to a previous cluster.
14. The method of claim 13 wherein the replica set comprises a plurality of replica members, and wherein the replica set is controlled and has consistent cluster operational data with respect to the previous cluster when a majority of replica members is exclusively reserved.
15. The method of claim 13 wherein attempting to obtain a right to exclusively reserve that replica member includes, executing a mutual exclusion algorithm.
16. The method of claim 13 wherein attempting to obtain a right to exclusively reserve that replica member includes, attempting to write a unique identifier to a location on the replica member, delaying, and reading from the location to determine whether the unique identifier is unchanged.
17. The method of claim 13 wherein arbitration is performed by challenging at the first node for ownership of the replica set when the first node does not represent the cluster.
18. The method of claim 13 further comprising, defending exclusive ownership of the replica set at the first node after control of the cluster is achieved.
19. The method of claim 13 further comprising determining which of the replica members of the replica set has cluster operational data that is most updated, and replicating that operational data to the other replica members of the replica set.
20. The method of claim 13 wherein arbitrating for each replica member includes breaking a reservation of the replica member by another node.
21. The method of claim 13 wherein arbitrating for each replica member includes, issuing a reset command for the replica member, delaying for a period of time, and issuing a reserve command for the replica member.
22. A computer-readable medium having computer-executable instructions, comprising:
representing a cluster by obtaining exclusive control of a majority of replica members in an available set of replica members;
detecting a status change of one replica member with respect to the available set; and
taking action in response to the changed status to ensure that the replica members are consistent with respect to any update logged thereto.
23. The computer-readable medium of claim 22 wherein taking action in response to the changed status includes running a recovery process to make the replica members consistent, including increasing an epoch number maintained on each available replica member.
24. The computer-readable medium of claim 22 wherein detecting a status change includes detecting that the one replica member is offline, and wherein taking action in response to the changed status includes determining whether a majority of replica members still exists, and if a majority of replica members does not still exist, preventing updates from being written to replica members that remain available.
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/203,489 US20060036896A1 (en) | 1999-03-26 | 2005-08-12 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US11/225,222 US7774469B2 (en) | 1999-03-26 | 2005-09-12 | Consistent cluster operational data in a server cluster using a quorum of replicas |
US12/815,373 US7984155B2 (en) | 1999-03-26 | 2010-06-14 | Consistent cluster operational data in a server cluster using a quorum of replicas |
US13/156,315 US8850007B2 (en) | 1999-03-26 | 2011-06-08 | Consistent cluster operational data in a server cluster using a quorum of replicas |
US13/156,305 US8850018B2 (en) | 1999-03-26 | 2011-06-08 | Consistent cluster operational data in a server cluster using a quorum of replicas |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/277,450 US6401120B1 (en) | 1999-03-26 | 1999-03-26 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US09/895,810 US6938084B2 (en) | 1999-03-26 | 2001-06-28 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US11/203,489 US20060036896A1 (en) | 1999-03-26 | 2005-08-12 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/895,810 Continuation US6938084B2 (en) | 1999-03-26 | 2001-06-28 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/225,222 Continuation-In-Part US7774469B2 (en) | 1999-03-26 | 2005-09-12 | Consistent cluster operational data in a server cluster using a quorum of replicas |
Publications (1)
Publication Number | Publication Date |
---|---|
US20060036896A1 true US20060036896A1 (en) | 2006-02-16 |
Family
ID=23060923
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/277,450 Expired - Lifetime US6401120B1 (en) | 1999-03-26 | 1999-03-26 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US09/895,810 Expired - Lifetime US6938084B2 (en) | 1999-03-26 | 2001-06-28 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US11/203,489 Abandoned US20060036896A1 (en) | 1999-03-26 | 2005-08-12 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/277,450 Expired - Lifetime US6401120B1 (en) | 1999-03-26 | 1999-03-26 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US09/895,810 Expired - Lifetime US6938084B2 (en) | 1999-03-26 | 2001-06-28 | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
Country Status (5)
Country | Link |
---|---|
US (3) | US6401120B1 (en) |
EP (1) | EP1222540B1 (en) |
AT (1) | ATE541259T1 (en) |
AU (1) | AU3729500A (en) |
WO (1) | WO2000058824A1 (en) |
Cited By (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20040078654A1 (en) * | 2002-03-29 | 2004-04-22 | Holland Mark C. | Hybrid quorum/primary-backup fault-tolerance model |
US20080313293A1 (en) * | 2001-09-06 | 2008-12-18 | Bea Systems, Inc. | System and method for exactly once message store communication |
US20090024868A1 (en) * | 2002-05-31 | 2009-01-22 | Joshi Darshan B | Business continuation policy for server consolidation environment |
US20090063892A1 (en) * | 2007-08-27 | 2009-03-05 | International Business Machines Corporation | Propogation by a controller of reservation made by a host for remote storage |
US20090063487A1 (en) * | 2007-08-29 | 2009-03-05 | International Business Machines Corporation | Securing transfer of ownership of a storage object from an unavailable owner node to another node |
US20090100436A1 (en) * | 2007-10-12 | 2009-04-16 | Microsoft Corporation | Partitioning system including a generic partitioning manager for partitioning resources |
US20090100441A1 (en) * | 2007-10-12 | 2009-04-16 | Microsoft Corporation | Resource assignment system with recovery notification |
US20100086753A1 (en) * | 2008-10-02 | 2010-04-08 | Wade Johnson | Foiled articles and methods of making same |
US20100232288A1 (en) * | 2009-03-10 | 2010-09-16 | Coatney Susan M | Takeover of a Failed Node of a Cluster Storage System on a Per Aggregate Basis |
US7900206B1 (en) | 2004-03-31 | 2011-03-01 | Symantec Operating Corporation | Information technology process workflow for data centers |
US20110178984A1 (en) * | 2010-01-18 | 2011-07-21 | Microsoft Corporation | Replication protocol for database systems |
US20110191299A1 (en) * | 2010-02-01 | 2011-08-04 | Microsoft Corporation | Logical data backup and rollback using incremental capture in a distributed database |
US20110238813A1 (en) * | 1999-03-26 | 2011-09-29 | Microsoft Corporation | Consistent cluster operational data in a server cluster using a quorum of replicas |
US20120023209A1 (en) * | 2010-07-20 | 2012-01-26 | Robert Adam Fletcher | Method and apparatus for scalable automated cluster control based on service level objectives to support applications requiring continuous availability |
US20120042202A1 (en) * | 2009-04-29 | 2012-02-16 | Thomas Rudolf Wenzel | Global write-log device for managing write logs of nodes of a cluster storage system |
US8145838B1 (en) | 2009-03-10 | 2012-03-27 | Netapp, Inc. | Processing and distributing write logs of nodes of a cluster storage system |
US8326990B1 (en) | 2005-07-15 | 2012-12-04 | Symantec Operating Corporation | Automated optimal workload balancing during failover in share-nothing database systems |
US8650281B1 (en) * | 2012-02-01 | 2014-02-11 | Symantec Corporation | Intelligent arbitration servers for network partition arbitration |
US20140215272A1 (en) * | 2013-01-31 | 2014-07-31 | Alcatel-Lucent Usa Inc. | Preventing extreme coresidency hazards in cloud computing |
US20150324416A1 (en) * | 2013-01-31 | 2015-11-12 | Hitachi, Ltd. | Management apparatus and management system |
WO2016010972A1 (en) * | 2014-07-17 | 2016-01-21 | Cohesity, Inc. | Dynamically changing members of a consensus group in a distributed self-healing coordination service |
WO2016108945A1 (en) * | 2014-12-30 | 2016-07-07 | Hewlett Packard Enterprise Development Lp | Cluster arbitration |
US20170177232A1 (en) * | 2015-12-16 | 2017-06-22 | International Business Machines Corporation | Repopulating failed replicas through modified consensus recovery |
US11256759B1 (en) | 2019-12-23 | 2022-02-22 | Lacework Inc. | Hierarchical graph analysis |
US20220334934A1 (en) * | 2021-04-16 | 2022-10-20 | EMC IP Holding Company LLC | Data storage cluster with quorum service protection |
US20220342769A1 (en) * | 2021-04-21 | 2022-10-27 | EMC IP Holding Company LLC | Application consistent network backup using three phase full quorum |
US20220398259A1 (en) * | 2020-12-23 | 2022-12-15 | Kuyun (Shanghai) Information Technology Co., Ltd. | Online analytical processing precomputation engine optimization method based on object storage and application |
US11637849B1 (en) | 2017-11-27 | 2023-04-25 | Lacework Inc. | Graph-based query composition |
US11770464B1 (en) | 2019-12-23 | 2023-09-26 | Lacework Inc. | Monitoring communications in a containerized environment |
US11792284B1 (en) | 2017-11-27 | 2023-10-17 | Lacework, Inc. | Using data transformations for monitoring a cloud compute environment |
US11831668B1 (en) | 2019-12-23 | 2023-11-28 | Lacework Inc. | Using a logical graph to model activity in a network environment |
US11909752B1 (en) | 2017-11-27 | 2024-02-20 | Lacework, Inc. | Detecting deviations from typical user behavior |
US11954130B1 (en) | 2019-12-23 | 2024-04-09 | Lacework Inc. | Alerting based on pod communication-based logical graph |
Families Citing this family (205)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6401120B1 (en) * | 1999-03-26 | 2002-06-04 | Microsoft Corporation | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US7756830B1 (en) | 1999-03-31 | 2010-07-13 | International Business Machines Corporation | Error detection protocol |
US6587860B1 (en) * | 1999-03-31 | 2003-07-01 | International Business Machines Corporation | Apparatus and method for tracking access to data resources in a cluster environment |
US7020695B1 (en) * | 1999-05-28 | 2006-03-28 | Oracle International Corporation | Using a cluster-wide shared repository to provide the latest consistent definition of the cluster (avoiding the partition-in time problem) |
US7076783B1 (en) | 1999-05-28 | 2006-07-11 | Oracle International Corporation | Providing figure of merit vote from application executing on a partitioned cluster |
US6871222B1 (en) | 1999-05-28 | 2005-03-22 | Oracle International Corporation | Quorumless cluster using disk-based messaging |
US6754661B1 (en) * | 1999-07-13 | 2004-06-22 | Microsoft Corporation | Hierarchical storage systems for holding evidentiary objects and methods of creating and operating upon hierarchical storage systems |
US6745240B1 (en) * | 1999-11-15 | 2004-06-01 | Ncr Corporation | Method and apparatus for configuring massively parallel systems |
US6769008B1 (en) * | 2000-01-10 | 2004-07-27 | Sun Microsystems, Inc. | Method and apparatus for dynamically altering configurations of clustered computer systems |
WO2001084338A2 (en) * | 2000-05-02 | 2001-11-08 | Sun Microsystems, Inc. | Cluster configuration repository |
WO2001082678A2 (en) * | 2000-05-02 | 2001-11-08 | Sun Microsystems, Inc. | Cluster membership monitor |
US7051097B1 (en) * | 2000-05-20 | 2006-05-23 | Ciena Corporation | Embedded database for computer system management |
US7325046B1 (en) * | 2000-05-31 | 2008-01-29 | International Business Machines Corporation | Method, system and program products for managing processing groups of a distributed computing environment |
US20020004849A1 (en) * | 2000-06-22 | 2002-01-10 | Elink Business | Fault tolerant internet communications system |
DE10038562B4 (en) * | 2000-08-03 | 2005-12-15 | Siemens Ag | System and method for transmitting data over data networks with data conversion by a COM car sounder |
US6968359B1 (en) * | 2000-08-14 | 2005-11-22 | International Business Machines Corporation | Merge protocol for clustered computer system |
US8027892B2 (en) * | 2001-03-28 | 2011-09-27 | International Business Machines Corporation | System and method for automating invoice processing with positive confirmation |
US7155403B2 (en) * | 2001-03-22 | 2006-12-26 | International Business Machines Corporation | System and method for leveraging procurement across companies and company groups |
US7197480B1 (en) * | 2000-09-07 | 2007-03-27 | International Business Machines Corporation | System and method for front end business logic and validation |
US6965938B1 (en) * | 2000-09-07 | 2005-11-15 | International Business Machines Corporation | System and method for clustering servers for performance and load balancing |
US7283976B2 (en) * | 2001-03-22 | 2007-10-16 | International Business Machines Corporation | System and method for invoice imaging through negative confirmation process |
US7356496B2 (en) * | 2001-03-22 | 2008-04-08 | International Business Machines Corporation | System and method for synchronizing ledger accounts by company group |
US7386495B2 (en) * | 2001-03-23 | 2008-06-10 | International Business Machines Corporation | System and method for processing tax codes by company group |
US7769844B2 (en) * | 2000-12-07 | 2010-08-03 | International Business Machines Corporation | Peer protocol status query in clustered computer system |
US7730213B2 (en) * | 2000-12-18 | 2010-06-01 | Oracle America, Inc. | Object-based storage device with improved reliability and fast crash recovery |
US6931450B2 (en) * | 2000-12-18 | 2005-08-16 | Sun Microsystems, Inc. | Direct access from client to storage device |
US7444335B1 (en) * | 2001-02-28 | 2008-10-28 | Oracle International Corporation | System and method for providing cooperative resource groups for high availability applications |
US7069317B1 (en) | 2001-02-28 | 2006-06-27 | Oracle International Corporation | System and method for providing out-of-band notification of service changes |
US6892205B1 (en) | 2001-02-28 | 2005-05-10 | Oracle International Corporation | System and method for pre-compiling a source cursor into a target library cache |
US7058629B1 (en) * | 2001-02-28 | 2006-06-06 | Oracle International Corporation | System and method for detecting termination of an application instance using locks |
US7243077B2 (en) * | 2001-03-02 | 2007-07-10 | International Business Machines Corporation | Method and computer program product for managing an internet trading network |
US20040205693A1 (en) * | 2001-04-23 | 2004-10-14 | Michael Alexander | Resource localization |
US7640582B2 (en) | 2003-04-16 | 2009-12-29 | Silicon Graphics International | Clustered filesystem for mix of trusted and untrusted nodes |
US20040139125A1 (en) * | 2001-06-05 | 2004-07-15 | Roger Strassburg | Snapshot copy of data volume during data access |
US6708175B2 (en) * | 2001-06-06 | 2004-03-16 | International Business Machines Corporation | Program support for disk fencing in a shared disk parallel file system across storage area network |
US7016946B2 (en) * | 2001-07-05 | 2006-03-21 | Sun Microsystems, Inc. | Method and system for establishing a quorum for a geographically distributed cluster of computers |
US20030014523A1 (en) * | 2001-07-13 | 2003-01-16 | John Teloh | Storage network data replicator |
US7127565B2 (en) * | 2001-08-20 | 2006-10-24 | Spinnaker Networks, Inc. | Method and system for safely arbitrating disk drive ownership using a timestamp voting algorithm |
US7231461B2 (en) * | 2001-09-14 | 2007-06-12 | International Business Machines Corporation | Synchronization of group state data when rejoining a member to a primary-backup group in a clustered computer system |
US7277952B2 (en) * | 2001-09-28 | 2007-10-02 | Microsoft Corporation | Distributed system resource protection via arbitration and ownership |
US20030084219A1 (en) * | 2001-10-26 | 2003-05-01 | Maxxan Systems, Inc. | System, apparatus and method for address forwarding for a computer network |
US6920469B2 (en) * | 2001-11-13 | 2005-07-19 | Tadpole Technology Plc | Unique ID management in disconnected database replication |
JP2005512190A (en) * | 2001-11-30 | 2005-04-28 | オラクル・インターナショナル・コーポレイション | Real composite objects that provide high availability of resources in networked systems |
DE60106467T2 (en) * | 2001-12-14 | 2006-02-23 | Hewlett-Packard Development Co., L.P., Houston | Procedure for installing monitoring agent, system and computer program of objects in an IT network monitoring |
US7145914B2 (en) | 2001-12-31 | 2006-12-05 | Maxxan Systems, Incorporated | System and method for controlling data paths of a network processor subsystem |
US7085846B2 (en) * | 2001-12-31 | 2006-08-01 | Maxxan Systems, Incorporated | Buffer to buffer credit flow control for computer network |
US7392421B1 (en) * | 2002-03-18 | 2008-06-24 | Symantec Operating Corporation | Framework for managing clustering and replication |
US7103884B2 (en) * | 2002-03-27 | 2006-09-05 | Lucent Technologies Inc. | Method for maintaining consistency and performing recovery in a replicated data storage system |
US7007047B2 (en) * | 2002-03-29 | 2006-02-28 | Panasas, Inc. | Internally consistent file system image in distributed object-based data storage |
US7379970B1 (en) * | 2002-04-05 | 2008-05-27 | Ciphermax, Inc. | Method and system for reduced distributed event handling in a network environment |
US7406038B1 (en) | 2002-04-05 | 2008-07-29 | Ciphermax, Incorporated | System and method for expansion of computer network switching system without disruption thereof |
US7307995B1 (en) | 2002-04-05 | 2007-12-11 | Ciphermax, Inc. | System and method for linking a plurality of network switches |
US20030200322A1 (en) * | 2002-04-18 | 2003-10-23 | International Business Machines Corporation | Autonomic system for selective administation isolation of a secure remote management of systems in a computer network |
US20030202510A1 (en) * | 2002-04-26 | 2003-10-30 | Maxxan Systems, Inc. | System and method for scalable switch fabric for computer network |
US20040006587A1 (en) * | 2002-07-02 | 2004-01-08 | Dell Products L.P. | Information handling system and method for clustering with internal cross coupled storage |
US20040030766A1 (en) * | 2002-08-12 | 2004-02-12 | Michael Witkowski | Method and apparatus for switch fabric configuration |
US7562156B2 (en) * | 2002-08-16 | 2009-07-14 | Symantec Operating Corporation | System and method for decoding communications between nodes of a cluster server |
US20040088401A1 (en) * | 2002-10-31 | 2004-05-06 | Ashutosh Tripathi | Method and apparatus for providing a highly available distributed event notification mechanism |
US8285825B1 (en) * | 2002-11-13 | 2012-10-09 | Novell, Inc. | Method and system for managing network resources based on a dynamic quorum |
US7739240B2 (en) * | 2002-12-09 | 2010-06-15 | Hewlett-Packard Development Company, L.P. | Replication and replica management in a wide area file system |
US6836798B1 (en) * | 2002-12-31 | 2004-12-28 | Sprint Communications Company, L.P. | Network model reconciliation using state analysis |
ES2229881B1 (en) * | 2003-02-27 | 2006-02-16 | Differend Games, S.A. | METHOD AND ELECTRONIC DEVICE FOR INTERACTIVE GAME CONTROL. |
US7389345B1 (en) | 2003-03-26 | 2008-06-17 | Sprint Communications Company L.P. | Filtering approach for network system alarms |
US7421493B1 (en) | 2003-04-28 | 2008-09-02 | Sprint Communications Company L.P. | Orphaned network resource recovery through targeted audit and reconciliation |
US7406694B2 (en) * | 2003-06-20 | 2008-07-29 | Microsoft Corporation | Method and system for tracking kernel resource usage |
US7603442B2 (en) * | 2003-06-20 | 2009-10-13 | Microsoft Corporation | Method and system for maintaining service dependency relationships in a computer system |
US7587475B2 (en) * | 2003-06-25 | 2009-09-08 | Nokia Siemens Networks Oy | System for joining a cluster by cloning configuration |
US7660833B2 (en) * | 2003-07-10 | 2010-02-09 | Microsoft Corporation | Granular control over the authority of replicated information via fencing and unfencing |
US7120821B1 (en) * | 2003-07-24 | 2006-10-10 | Unisys Corporation | Method to revive and reconstitute majority node set clusters |
US7490113B2 (en) * | 2003-08-27 | 2009-02-10 | International Business Machines Corporation | Database log capture that publishes transactions to multiple targets to handle unavailable targets by separating the publishing of subscriptions and subsequently recombining the publishing |
US7562123B2 (en) * | 2003-09-05 | 2009-07-14 | Palo Alto Research Center Incorporated | Systems and methods for distributed group formation and maintenance in geographically based networks |
US7266815B2 (en) * | 2003-09-29 | 2007-09-04 | International Business Machines Corporation | Automated control of a licensed internal code update on a storage controller |
US7392420B2 (en) * | 2003-09-29 | 2008-06-24 | International Business Machines Corporation | Automated error recovery of a licensed internal code update on a storage controller |
US7076622B2 (en) * | 2003-09-30 | 2006-07-11 | International Business Machines Corporation | System and method for detecting and sharing common blocks in an object storage system |
US8060619B1 (en) * | 2003-11-07 | 2011-11-15 | Symantec Operating Corporation | Direct connections to a plurality of storage object replicas in a computer network |
US20050108389A1 (en) * | 2003-11-13 | 2005-05-19 | International Business Machines Corporation | Network endpoint health check |
US20050132379A1 (en) * | 2003-12-11 | 2005-06-16 | Dell Products L.P. | Method, system and software for allocating information handling system resources in response to high availability cluster fail-over events |
US8499078B2 (en) * | 2003-12-24 | 2013-07-30 | Sap Aktiengesellschaft | Address generation in distributed systems using tree method |
US8103772B2 (en) * | 2003-12-24 | 2012-01-24 | Sap Aktiengesellschaft | Cluster extension in distributed systems using tree method |
US7584271B2 (en) * | 2004-01-07 | 2009-09-01 | International Business Machines Corporation | Method, system, and computer readable medium for delaying the configuration of a shared resource |
JP4516322B2 (en) * | 2004-01-28 | 2010-08-04 | 株式会社日立製作所 | A computer system having a shared exclusive control method between sites having a storage system shared by a plurality of host devices |
US7457248B1 (en) * | 2004-02-10 | 2008-11-25 | Cisco Technology, Inc. | Graceful shutdown of network resources in data networks |
US7231541B1 (en) | 2004-02-17 | 2007-06-12 | Unisys Corporation | Method for predictably feeding and consuming RAM memory to recognize over-consumption of resources |
US7814064B2 (en) * | 2004-05-12 | 2010-10-12 | Oracle International Corporation | Dynamic distributed consensus algorithm |
US7991753B2 (en) * | 2004-05-21 | 2011-08-02 | International Business Machines Corporation | Lock acquisition among nodes of divided cluster |
US7523204B2 (en) * | 2004-06-01 | 2009-04-21 | International Business Machines Corporation | Coordinated quiesce of a distributed file system |
US7334154B2 (en) * | 2004-06-18 | 2008-02-19 | Microsoft Corporation | Efficient changing of replica sets in distributed fault-tolerant computing system |
JP2006031335A (en) * | 2004-07-15 | 2006-02-02 | Hitachi Ltd | Information processing system and method |
US7277990B2 (en) | 2004-09-30 | 2007-10-02 | Sanjeev Jain | Method and apparatus providing efficient queue descriptor memory access |
US20080288659A1 (en) * | 2006-11-09 | 2008-11-20 | Microsoft Corporation | Maintaining consistency within a federation infrastructure |
US20060100981A1 (en) * | 2004-11-04 | 2006-05-11 | International Business Machines Corporation | Apparatus and method for quorum-based power-down of unresponsive servers in a computer cluster |
US7555630B2 (en) | 2004-12-21 | 2009-06-30 | Intel Corporation | Method and apparatus to provide efficient communication between multi-threaded processing elements in a processor unit |
US7418543B2 (en) | 2004-12-21 | 2008-08-26 | Intel Corporation | Processor having content addressable memory with command ordering |
US7467256B2 (en) | 2004-12-28 | 2008-12-16 | Intel Corporation | Processor having content addressable memory for block-based queue structures |
US8364633B2 (en) * | 2005-01-12 | 2013-01-29 | Wandisco, Inc. | Distributed computing systems and system components thereof |
US9424272B2 (en) | 2005-01-12 | 2016-08-23 | Wandisco, Inc. | Distributed file system using consensus nodes |
US9332069B2 (en) | 2012-12-28 | 2016-05-03 | Wandisco, Inc. | Methods, devices and systems for initiating, forming and joining memberships in distributed computing systems |
US9361311B2 (en) | 2005-01-12 | 2016-06-07 | Wandisco, Inc. | Distributed file system using consensus nodes |
US20060156055A1 (en) * | 2005-01-13 | 2006-07-13 | Dell Products L.P. | Storage network that includes an arbiter for managing access to storage resources |
JP2006195821A (en) * | 2005-01-14 | 2006-07-27 | Fujitsu Ltd | Method for controlling information processing system, information processing system, direct memory access controller, and program |
US8185905B2 (en) * | 2005-03-18 | 2012-05-22 | International Business Machines Corporation | Resource allocation in computing systems according to permissible flexibilities in the recommended resource requirements |
US7587623B2 (en) * | 2005-05-02 | 2009-09-08 | Cinnober Financial Technology Ab | Systems and methods for recovering a trading system |
EP1746507B1 (en) * | 2005-05-02 | 2010-01-20 | Cinnober Financial Technology AB | System and method for recovering a trading system |
US7739677B1 (en) * | 2005-05-27 | 2010-06-15 | Symantec Operating Corporation | System and method to prevent data corruption due to split brain in shared data clusters |
US20060294413A1 (en) * | 2005-06-28 | 2006-12-28 | Filz Frank S | Fault tolerant rolling software upgrade in a cluster |
US7870230B2 (en) * | 2005-07-15 | 2011-01-11 | International Business Machines Corporation | Policy-based cluster quorum determination |
US7464121B2 (en) * | 2006-01-06 | 2008-12-09 | International Business Machines Corporation | Apparatus for sending a sequence of asynchronous messages to the same member of a clustered consumer |
US7760743B2 (en) * | 2006-03-06 | 2010-07-20 | Oracle America, Inc. | Effective high availability cluster management and effective state propagation for failure recovery in high availability clusters |
US7886065B1 (en) * | 2006-03-28 | 2011-02-08 | Symantec Corporation | Detecting reboot events to enable NAC reassessment |
US7644305B2 (en) | 2006-05-18 | 2010-01-05 | Microsoft Corporation | Quorum establishment based on a vote from a voting device |
US8429396B1 (en) * | 2006-05-31 | 2013-04-23 | Juniper Networks, Inc. | Peer discovery and secure communication in failover schemes |
US8209405B1 (en) | 2006-06-12 | 2012-06-26 | Juniper Networks, Inc. | Failover scheme with service-based segregation |
US8255675B1 (en) | 2006-06-30 | 2012-08-28 | Symantec Operating Corporation | System and method for storage management of file system configuration data |
US20080065704A1 (en) * | 2006-09-12 | 2008-03-13 | Microsoft Corporation | Data and replica placement using r-out-of-k hash functions |
US7610429B2 (en) * | 2007-01-30 | 2009-10-27 | Hewlett-Packard Development Company, L.P. | Method and system for determining device criticality in a computer configuration |
US8209417B2 (en) * | 2007-03-08 | 2012-06-26 | Oracle International Corporation | Dynamic resource profiles for clusterware-managed resources |
US7685179B2 (en) * | 2007-03-13 | 2010-03-23 | Microsoft Corporation | Network flow for constrained replica placement |
JP2008234445A (en) * | 2007-03-22 | 2008-10-02 | Brother Ind Ltd | Content distributed storage system, duplicate data acquisition method, node device, and node processing program |
US7975109B2 (en) | 2007-05-30 | 2011-07-05 | Schooner Information Technology, Inc. | System including a fine-grained memory and a less-fine-grained memory |
US7788522B1 (en) * | 2007-05-31 | 2010-08-31 | Oracle America, Inc. | Autonomous cluster organization, collision detection, and resolutions |
US8918490B1 (en) | 2007-07-12 | 2014-12-23 | Oracle America Inc. | Locality and time based dependency relationships in clusters |
US8856206B2 (en) * | 2007-08-28 | 2014-10-07 | International Business Machines Corporation | Maintaining message versions at nodes in a network |
US20090132716A1 (en) * | 2007-11-15 | 2009-05-21 | Junqueira Flavio P | Fault-tolerant distributed services methods and systems |
US9027030B2 (en) | 2007-11-29 | 2015-05-05 | Red Hat, Inc. | Commit-one-phase distributed transactions with multiple starting participants |
US7849223B2 (en) * | 2007-12-07 | 2010-12-07 | Microsoft Corporation | Virtually synchronous Paxos |
US8229945B2 (en) | 2008-03-20 | 2012-07-24 | Schooner Information Technology, Inc. | Scalable database management software on a cluster of nodes using a shared-distributed flash memory |
US8732386B2 (en) * | 2008-03-20 | 2014-05-20 | Sandisk Enterprise IP LLC. | Sharing data fabric for coherent-distributed caching of multi-node shared-distributed flash memory |
JP5374900B2 (en) * | 2008-03-27 | 2013-12-25 | 富士通株式会社 | Computer system backup method |
US8352421B2 (en) * | 2008-05-28 | 2013-01-08 | Red Hat, Inc. | Recording distributed transactions using probabalistic data structures |
US7543046B1 (en) | 2008-05-30 | 2009-06-02 | International Business Machines Corporation | Method for managing cluster node-specific quorum roles |
US7962780B2 (en) * | 2008-08-22 | 2011-06-14 | International Business Machines Corporation | Command sequence numbering apparatus and method |
US8543773B2 (en) * | 2008-08-25 | 2013-09-24 | International Business Machines Corporation | Distributed shared memory |
SE533007C2 (en) * | 2008-10-24 | 2010-06-08 | Ilt Productions Ab | Distributed data storage |
US8725951B2 (en) | 2010-04-12 | 2014-05-13 | Sandisk Enterprise Ip Llc | Efficient flash memory-based object store |
US9164554B2 (en) | 2010-04-12 | 2015-10-20 | Sandisk Enterprise Ip Llc | Non-volatile solid-state storage system supporting high bandwidth and random access |
US8868487B2 (en) | 2010-04-12 | 2014-10-21 | Sandisk Enterprise Ip Llc | Event processing in a flash memory-based object store |
US8856593B2 (en) | 2010-04-12 | 2014-10-07 | Sandisk Enterprise Ip Llc | Failure recovery using consensus replication in a distributed flash memory system |
US9047351B2 (en) | 2010-04-12 | 2015-06-02 | Sandisk Enterprise Ip Llc | Cluster of processing nodes with distributed global flash memory using commodity server technology |
US11726955B2 (en) | 2010-06-19 | 2023-08-15 | Hewlett Packard Enterprise Development Lp | Methods and apparatus for efficient container location database snapshot operation |
US9323775B2 (en) * | 2010-06-19 | 2016-04-26 | Mapr Technologies, Inc. | Map-reduce ready distributed file system |
US8954385B2 (en) | 2010-06-28 | 2015-02-10 | Sandisk Enterprise Ip Llc | Efficient recovery of transactional data stores |
US8356072B1 (en) * | 2010-07-14 | 2013-01-15 | Netapp, Inc. | Systems and methods for scalable heterogeneous and homogeneous unified enterprise storage clusters |
US20120124238A1 (en) * | 2010-11-12 | 2012-05-17 | Alcatel-Lucent Bell N.V. | Prioritization of routing information updates |
US8949566B2 (en) | 2010-12-02 | 2015-02-03 | International Business Machines Corporation | Locking access to data storage shared by a plurality of compute nodes |
US8694733B2 (en) | 2011-01-03 | 2014-04-08 | Sandisk Enterprise Ip Llc | Slave consistency in a synchronous replication environment |
US8326800B2 (en) * | 2011-03-18 | 2012-12-04 | Microsoft Corporation | Seamless upgrades in a distributed database system |
US8874515B2 (en) | 2011-04-11 | 2014-10-28 | Sandisk Enterprise Ip Llc | Low level object version tracking using non-volatile memory write generations |
US9009106B1 (en) | 2011-08-10 | 2015-04-14 | Nutanix, Inc. | Method and system for implementing writable snapshots in a virtualized storage environment |
US8850130B1 (en) | 2011-08-10 | 2014-09-30 | Nutanix, Inc. | Metadata for managing I/O and storage for a virtualization |
US8863124B1 (en) | 2011-08-10 | 2014-10-14 | Nutanix, Inc. | Architecture for managing I/O and storage for a virtualization environment |
US9747287B1 (en) * | 2011-08-10 | 2017-08-29 | Nutanix, Inc. | Method and system for managing metadata for a virtualization environment |
US8549518B1 (en) | 2011-08-10 | 2013-10-01 | Nutanix, Inc. | Method and system for implementing a maintenanece service for managing I/O and storage for virtualization environment |
US9652265B1 (en) | 2011-08-10 | 2017-05-16 | Nutanix, Inc. | Architecture for managing I/O and storage for a virtualization environment with multiple hypervisor types |
US8601473B1 (en) | 2011-08-10 | 2013-12-03 | Nutanix, Inc. | Architecture for managing I/O and storage for a virtualization environment |
US8769138B2 (en) | 2011-09-02 | 2014-07-01 | Compuverde Ab | Method for data retrieval from a distributed data storage system |
US8719417B1 (en) * | 2011-10-14 | 2014-05-06 | Google Inc. | Resource allocation in distributed systems |
US9069827B1 (en) * | 2012-01-17 | 2015-06-30 | Amazon Technologies, Inc. | System and method for adjusting membership of a data replication group |
US9489434B1 (en) * | 2012-01-17 | 2016-11-08 | Amazon Technologies, Inc. | System and method for replication log branching avoidance using post-failover rejoin |
US8930312B1 (en) * | 2012-01-17 | 2015-01-06 | Amazon Technologies, Inc. | System and method for splitting a replicated data partition |
US9116862B1 (en) | 2012-01-17 | 2015-08-25 | Amazon Technologies, Inc. | System and method for data replication using a single master failover protocol |
US8843441B1 (en) | 2012-01-17 | 2014-09-23 | Amazon Technologies, Inc. | System and method for maintaining a master replica for reads and writes in a data store |
US10169097B2 (en) | 2012-01-23 | 2019-01-01 | Microsoft Technology Licensing, Llc | Dynamic quorum for distributed systems |
US8447730B1 (en) * | 2012-01-31 | 2013-05-21 | Yahoo! Inc. | Probe system for replication monitoring |
US9135064B2 (en) | 2012-03-07 | 2015-09-15 | Sandisk Enterprise Ip Llc | Fine grained adaptive throttling of background processes |
US9772866B1 (en) | 2012-07-17 | 2017-09-26 | Nutanix, Inc. | Architecture for implementing a virtualization environment and appliance |
US20140075170A1 (en) * | 2012-09-12 | 2014-03-13 | International Business Machines Corporation | Automated firmware voting to enable multi-enclosure federated systems |
US8838535B2 (en) * | 2012-10-05 | 2014-09-16 | Oracle International Corporation | Providing services across systems that manage distributed replicas |
US9264516B2 (en) | 2012-12-28 | 2016-02-16 | Wandisco, Inc. | Methods, devices and systems enabling a secure and authorized induction of a node into a group of nodes in a distributed computing environment |
US9448839B2 (en) | 2013-03-08 | 2016-09-20 | Oracle International Corporation | Backoff job queue polling mechanism |
US9009215B2 (en) | 2013-03-15 | 2015-04-14 | Wandisco, Inc. | Methods, devices and systems for dynamically managing memberships in replicated state machines within a distributed computing environment |
US9396076B2 (en) * | 2013-06-07 | 2016-07-19 | Oracle International Corporation | Centralized version control system having high availability |
ES2881606T3 (en) | 2014-03-31 | 2021-11-30 | Wandisco Inc | Geographically distributed file system using coordinated namespace replication |
US9686206B2 (en) * | 2014-04-29 | 2017-06-20 | Silicon Graphics International Corp. | Temporal based collaborative mutual exclusion control of a shared resource |
US10803012B1 (en) * | 2014-05-09 | 2020-10-13 | Amazon Technologies, Inc. | Variable data replication for storage systems implementing quorum-based durability schemes |
US9742692B2 (en) * | 2014-06-23 | 2017-08-22 | Microsoft Technology Licensing, Llc | Acquiring resource lease using multiple lease servers |
US9723064B1 (en) | 2014-09-02 | 2017-08-01 | Amazon Technologies, Inc. | Hybrid quorum policies for durable consensus in distributed systems |
CA2969210C (en) * | 2014-12-01 | 2024-02-20 | Informatica Llc | Method, apparatus, and comuter-readable medium for processing a message by a message broker system |
GB2554250B (en) | 2015-07-02 | 2021-09-01 | Google Llc | Distributed storage system with replica location selection |
US9898378B2 (en) * | 2015-11-10 | 2018-02-20 | International Business Machines Corporation | Smart selection of a storage module to be excluded |
US10733201B1 (en) | 2015-11-30 | 2020-08-04 | Amazon Technologies, Inc. | Dynamic provisioning for data replication groups |
US11640410B1 (en) | 2015-12-02 | 2023-05-02 | Amazon Technologies, Inc. | Distributed log processing for data replication groups |
US10567499B1 (en) | 2015-12-02 | 2020-02-18 | Amazon Technologies, Inc. | Unsupervised round robin catch up algorithm |
US10924543B1 (en) | 2015-12-18 | 2021-02-16 | Amazon Technologies, Inc. | Deployment strategy for maintaining integrity of replication groups |
US10467103B1 (en) | 2016-03-25 | 2019-11-05 | Nutanix, Inc. | Efficient change block training |
US10055315B2 (en) | 2016-06-29 | 2018-08-21 | Microsoft Technology Licensing, Llc | Failure monitoring in distributed computing systems |
US10114712B2 (en) * | 2016-06-29 | 2018-10-30 | Microsoft Technology Licensing, Llc | Failure detection via implicit leases in distributed computing systems |
US11075806B1 (en) | 2016-06-30 | 2021-07-27 | Juniper Networks, Inc. | Hierarchical naming scheme for state propagation within network devices |
US10521311B1 (en) | 2016-06-30 | 2019-12-31 | Amazon Technologies, Inc. | Prioritized leadership for data replication groups |
US10565227B1 (en) | 2016-08-31 | 2020-02-18 | Amazon Technologies, Inc. | Leadership lease protocol for data replication groups |
US10887173B2 (en) | 2016-12-21 | 2021-01-05 | Juniper Networks, Inc. | Communicating state information in distributed operating systems |
US11316744B2 (en) | 2016-12-21 | 2022-04-26 | Juniper Networks, Inc. | Organizing execution of distributed operating systems for network devices |
US11316775B2 (en) * | 2016-12-21 | 2022-04-26 | Juniper Networks, Inc. | Maintaining coherency in distributed operating systems for network devices |
US11360942B2 (en) | 2017-03-13 | 2022-06-14 | Wandisco Inc. | Methods, devices and systems for maintaining consistency of metadata and data across data centers |
US10628057B2 (en) * | 2017-03-28 | 2020-04-21 | Hewlett Packard Enterprise Development Lp | Capability based locking and access of shared persistent memory |
US10331523B2 (en) | 2017-04-04 | 2019-06-25 | International Business Machines Corporation | Recovering a failed clustered system using configuration data fragments |
US10341168B2 (en) * | 2017-04-18 | 2019-07-02 | Sap Se | Topology manager for failure detection in a distributed computing system |
JP6934754B2 (en) * | 2017-06-15 | 2021-09-15 | 株式会社日立製作所 | Distributed processing system, distributed processing system management method, and distributed processing system management program |
US10742724B2 (en) * | 2017-08-17 | 2020-08-11 | Hewlett Packard Enterprise Development Lp | Cluster computer system with failover handling |
US10789267B1 (en) | 2017-09-21 | 2020-09-29 | Amazon Technologies, Inc. | Replication group data management |
US11108857B2 (en) | 2018-02-27 | 2021-08-31 | Elasticsearch B.V. | Self-replicating management services for distributed computing architectures |
US11019166B2 (en) * | 2018-02-27 | 2021-05-25 | Elasticsearch B.V. | Management services for distributed computing architectures using rolling changes |
US11016679B2 (en) | 2018-06-29 | 2021-05-25 | Seagate Technology Llc | Balanced die set execution in a data storage system |
US11095742B2 (en) | 2019-03-27 | 2021-08-17 | Juniper Networks, Inc. | Query proxy for delivery of dynamic system state |
US11372818B2 (en) * | 2020-01-07 | 2022-06-28 | Samsung Electronics Co., Ltd. | Scaling performance for large scale replica sets for a strongly consistent distributed system |
US11397545B1 (en) | 2021-01-20 | 2022-07-26 | Pure Storage, Inc. | Emulating persistent reservations in a cloud-based storage system |
Citations (26)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5553239A (en) * | 1994-11-10 | 1996-09-03 | At&T Corporation | Management facility for server entry and application utilization in a multi-node server configuration |
US5673384A (en) * | 1995-10-06 | 1997-09-30 | Hewlett-Packard Company | Dual disk lock arbitration between equal sized partition of a cluster |
US5727206A (en) * | 1996-07-31 | 1998-03-10 | Ncr Corporation | On-line file system correction within a clustered processing system |
US5754821A (en) * | 1993-12-23 | 1998-05-19 | International Business Machines Corporation | Method and system for providing access to a protected partition of a memory device utilizing a passthru command |
US5781910A (en) * | 1996-09-13 | 1998-07-14 | Stratus Computer, Inc. | Preforming concurrent transactions in a replicated database environment |
US5828876A (en) * | 1996-07-31 | 1998-10-27 | Ncr Corporation | File system for a clustered processing system |
US5828889A (en) * | 1996-05-31 | 1998-10-27 | Sun Microsystems, Inc. | Quorum mechanism in a two-node distributed computer system |
US5835784A (en) * | 1995-06-15 | 1998-11-10 | Intel Corporation | System for booting processor from remote memory by preventing host processor from configuring an environment of processor while configuring an interface unit between processor and remote memory |
US5893086A (en) * | 1997-07-11 | 1999-04-06 | International Business Machines Corporation | Parallel file system and method with extensible hashing |
US5909540A (en) * | 1996-11-22 | 1999-06-01 | Mangosoft Corporation | System and method for providing highly available data storage using globally addressable memory |
US5917998A (en) * | 1996-07-26 | 1999-06-29 | International Business Machines Corporation | Method and apparatus for establishing and maintaining the status of membership sets used in mirrored read and write input/output without logging |
US5927050A (en) * | 1997-11-15 | 1999-07-27 | International Business Machines Corporation | In-box configuration method for a computer system |
US5940838A (en) * | 1997-07-11 | 1999-08-17 | International Business Machines Corporation | Parallel file system and method anticipating cache usage patterns |
US6014669A (en) * | 1997-10-01 | 2000-01-11 | Sun Microsystems, Inc. | Highly-available distributed cluster configuration database |
US6061740A (en) * | 1996-12-09 | 2000-05-09 | Novell, Inc. | Method and apparatus for heterogeneous network management |
US6108781A (en) * | 1997-07-31 | 2000-08-22 | Intel Corporation | Bootstrap processor selection architecture in SMP system |
US6279032B1 (en) * | 1997-11-03 | 2001-08-21 | Microsoft Corporation | Method and system for quorum resource arbitration in a server cluster |
US6301462B1 (en) * | 1999-01-15 | 2001-10-09 | Unext. Com | Online collaborative apprenticeship |
US6314526B1 (en) * | 1998-07-10 | 2001-11-06 | International Business Machines Corporation | Resource group quorum scheme for highly scalable and highly available cluster system management |
US6401120B1 (en) * | 1999-03-26 | 2002-06-04 | Microsoft Corporation | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US6662219B1 (en) * | 1999-12-15 | 2003-12-09 | Microsoft Corporation | System for determining at subgroup of nodes relative weight to represent cluster by obtaining exclusive possession of quorum resource |
US6681251B1 (en) * | 1999-11-18 | 2004-01-20 | International Business Machines Corporation | Workload balancing in clustered application servers |
US6691139B2 (en) * | 2001-01-31 | 2004-02-10 | Hewlett-Packard Development Co., Ltd. | Recreation of archives at a disaster recovery site |
US6701453B2 (en) * | 1997-05-13 | 2004-03-02 | Micron Technology, Inc. | System for clustering software applications |
US6738872B2 (en) * | 2000-12-22 | 2004-05-18 | International Business Machines Corporation | Clustered computer system with deadlock avoidance |
US6807557B1 (en) * | 2000-05-31 | 2004-10-19 | International Business Machines Corporation | Method, system and program products for providing clusters of a computing environment |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5146568A (en) | 1988-09-06 | 1992-09-08 | Digital Equipment Corporation | Remote bootstrapping a node over communication link by initially requesting remote storage access program which emulates local disk to load other programs |
DE69222528T2 (en) | 1991-06-26 | 1998-04-16 | Ast Research Inc | DISTRIBUTED MULTIPROCESSOR INITIALIZATION AND SELF-TESTING SYSTEM |
US5404527A (en) * | 1992-12-31 | 1995-04-04 | Unisys Corporation | System and method for remote program load |
US5696895A (en) | 1995-05-19 | 1997-12-09 | Compaq Computer Corporation | Fault tolerant multiple network servers |
US5666486A (en) * | 1995-06-23 | 1997-09-09 | Data General Corporation | Multiprocessor cluster membership manager framework |
US5996075A (en) | 1995-11-02 | 1999-11-30 | Sun Microsystems, Inc. | Method and apparatus for reliable disk fencing in a multicomputer system |
US6148377A (en) * | 1996-11-22 | 2000-11-14 | Mangosoft Corporation | Shared memory computer networks |
US5892913A (en) * | 1996-12-02 | 1999-04-06 | International Business Machines Corporation | System and method for datastreams employing shared loop architecture multimedia subsystem clusters |
US6108699A (en) * | 1997-06-27 | 2000-08-22 | Sun Microsystems, Inc. | System and method for modifying membership in a clustered distributed computer system and updating system configuration |
US5946686A (en) | 1997-07-11 | 1999-08-31 | International Business Machines Corporation | Parallel file system and method with quota allocation |
US5999712A (en) | 1997-10-21 | 1999-12-07 | Sun Microsystems, Inc. | Determining cluster membership in a distributed computer system |
US5974547A (en) * | 1998-03-20 | 1999-10-26 | 3Com Corporation | Technique for reliable network booting of an operating system to a client computer |
US6360331B2 (en) * | 1998-04-17 | 2002-03-19 | Microsoft Corporation | Method and system for transparently failing over application configuration information in a server cluster |
US6311217B1 (en) * | 1998-06-04 | 2001-10-30 | Compaq Computer Corporation | Method and apparatus for improved cluster administration |
US6286056B1 (en) * | 1998-06-26 | 2001-09-04 | Seagate Technology Llc | Data storage device with small computer system interface providing persistent reservations |
US6665304B2 (en) | 1998-12-31 | 2003-12-16 | Hewlett-Packard Development Company, L.P. | Method and apparatus for providing an integrated cluster alias address |
US6363495B1 (en) * | 1999-01-19 | 2002-03-26 | International Business Machines Corporation | Method and apparatus for partition resolution in clustered computer systems |
US6438705B1 (en) * | 1999-01-29 | 2002-08-20 | International Business Machines Corporation | Method and apparatus for building and managing multi-clustered computer systems |
US6463532B1 (en) * | 1999-02-23 | 2002-10-08 | Compaq Computer Corporation | System and method for effectuating distributed consensus among members of a processor set in a multiprocessor computing system through the use of shared storage resources |
US6487622B1 (en) * | 1999-10-28 | 2002-11-26 | Ncr Corporation | Quorum arbitrator for a high availability system |
US6553387B1 (en) * | 1999-11-29 | 2003-04-22 | Microsoft Corporation | Logical volume configuration data management determines whether to expose the logical volume on-line, off-line request based on comparison of volume epoch numbers on each extents of the volume identifiers |
US6654902B1 (en) * | 2000-04-11 | 2003-11-25 | Hewlett-Packard Development Company, L.P. | Persistent reservation IO barriers |
US6701332B1 (en) * | 2000-07-31 | 2004-03-02 | Hewlett-Packard Development Company, L.P. | Cluster file system multi-volume root support |
US6950945B2 (en) * | 2001-06-21 | 2005-09-27 | International Business Machines Corporation | Apparatus and method for intersystem lock optimization |
-
1999
- 1999-03-26 US US09/277,450 patent/US6401120B1/en not_active Expired - Lifetime
-
2000
- 2000-03-06 AU AU37295/00A patent/AU3729500A/en not_active Abandoned
- 2000-03-06 AT AT00916146T patent/ATE541259T1/en active
- 2000-03-06 EP EP00916146A patent/EP1222540B1/en not_active Expired - Lifetime
- 2000-03-06 WO PCT/US2000/005977 patent/WO2000058824A1/en active Application Filing
-
2001
- 2001-06-28 US US09/895,810 patent/US6938084B2/en not_active Expired - Lifetime
-
2005
- 2005-08-12 US US11/203,489 patent/US20060036896A1/en not_active Abandoned
Patent Citations (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5754821A (en) * | 1993-12-23 | 1998-05-19 | International Business Machines Corporation | Method and system for providing access to a protected partition of a memory device utilizing a passthru command |
US5553239A (en) * | 1994-11-10 | 1996-09-03 | At&T Corporation | Management facility for server entry and application utilization in a multi-node server configuration |
US5835784A (en) * | 1995-06-15 | 1998-11-10 | Intel Corporation | System for booting processor from remote memory by preventing host processor from configuring an environment of processor while configuring an interface unit between processor and remote memory |
US5673384A (en) * | 1995-10-06 | 1997-09-30 | Hewlett-Packard Company | Dual disk lock arbitration between equal sized partition of a cluster |
US5828889A (en) * | 1996-05-31 | 1998-10-27 | Sun Microsystems, Inc. | Quorum mechanism in a two-node distributed computer system |
US5917998A (en) * | 1996-07-26 | 1999-06-29 | International Business Machines Corporation | Method and apparatus for establishing and maintaining the status of membership sets used in mirrored read and write input/output without logging |
US5727206A (en) * | 1996-07-31 | 1998-03-10 | Ncr Corporation | On-line file system correction within a clustered processing system |
US5828876A (en) * | 1996-07-31 | 1998-10-27 | Ncr Corporation | File system for a clustered processing system |
US5781910A (en) * | 1996-09-13 | 1998-07-14 | Stratus Computer, Inc. | Preforming concurrent transactions in a replicated database environment |
US5909540A (en) * | 1996-11-22 | 1999-06-01 | Mangosoft Corporation | System and method for providing highly available data storage using globally addressable memory |
US6061740A (en) * | 1996-12-09 | 2000-05-09 | Novell, Inc. | Method and apparatus for heterogeneous network management |
US6701453B2 (en) * | 1997-05-13 | 2004-03-02 | Micron Technology, Inc. | System for clustering software applications |
US5940838A (en) * | 1997-07-11 | 1999-08-17 | International Business Machines Corporation | Parallel file system and method anticipating cache usage patterns |
US5893086A (en) * | 1997-07-11 | 1999-04-06 | International Business Machines Corporation | Parallel file system and method with extensible hashing |
US6108781A (en) * | 1997-07-31 | 2000-08-22 | Intel Corporation | Bootstrap processor selection architecture in SMP system |
US6014669A (en) * | 1997-10-01 | 2000-01-11 | Sun Microsystems, Inc. | Highly-available distributed cluster configuration database |
US6279032B1 (en) * | 1997-11-03 | 2001-08-21 | Microsoft Corporation | Method and system for quorum resource arbitration in a server cluster |
US5927050A (en) * | 1997-11-15 | 1999-07-27 | International Business Machines Corporation | In-box configuration method for a computer system |
US6314526B1 (en) * | 1998-07-10 | 2001-11-06 | International Business Machines Corporation | Resource group quorum scheme for highly scalable and highly available cluster system management |
US6301462B1 (en) * | 1999-01-15 | 2001-10-09 | Unext. Com | Online collaborative apprenticeship |
US6401120B1 (en) * | 1999-03-26 | 2002-06-04 | Microsoft Corporation | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US6938084B2 (en) * | 1999-03-26 | 2005-08-30 | Microsoft Corporation | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas |
US6681251B1 (en) * | 1999-11-18 | 2004-01-20 | International Business Machines Corporation | Workload balancing in clustered application servers |
US6662219B1 (en) * | 1999-12-15 | 2003-12-09 | Microsoft Corporation | System for determining at subgroup of nodes relative weight to represent cluster by obtaining exclusive possession of quorum resource |
US6807557B1 (en) * | 2000-05-31 | 2004-10-19 | International Business Machines Corporation | Method, system and program products for providing clusters of a computing environment |
US6738872B2 (en) * | 2000-12-22 | 2004-05-18 | International Business Machines Corporation | Clustered computer system with deadlock avoidance |
US6691139B2 (en) * | 2001-01-31 | 2004-02-10 | Hewlett-Packard Development Co., Ltd. | Recreation of archives at a disaster recovery site |
Cited By (55)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8850007B2 (en) | 1999-03-26 | 2014-09-30 | Microsoft Corporation | Consistent cluster operational data in a server cluster using a quorum of replicas |
US8850018B2 (en) | 1999-03-26 | 2014-09-30 | Microsoft Corporation | Consistent cluster operational data in a server cluster using a quorum of replicas |
US20110238813A1 (en) * | 1999-03-26 | 2011-09-29 | Microsoft Corporation | Consistent cluster operational data in a server cluster using a quorum of replicas |
US20110238842A1 (en) * | 1999-03-26 | 2011-09-29 | Microsoft Corporation | Consistent cluster operational data in a server cluster using a quorum of replicas |
US7921169B2 (en) * | 2001-09-06 | 2011-04-05 | Oracle International Corporation | System and method for exactly once message store communication |
US20080313293A1 (en) * | 2001-09-06 | 2008-12-18 | Bea Systems, Inc. | System and method for exactly once message store communication |
US20040078654A1 (en) * | 2002-03-29 | 2004-04-22 | Holland Mark C. | Hybrid quorum/primary-backup fault-tolerance model |
US7191357B2 (en) * | 2002-03-29 | 2007-03-13 | Panasas, Inc. | Hybrid quorum/primary-backup fault-tolerance model |
US7529822B2 (en) * | 2002-05-31 | 2009-05-05 | Symantec Operating Corporation | Business continuation policy for server consolidation environment |
US20090024868A1 (en) * | 2002-05-31 | 2009-01-22 | Joshi Darshan B | Business continuation policy for server consolidation environment |
US7900206B1 (en) | 2004-03-31 | 2011-03-01 | Symantec Operating Corporation | Information technology process workflow for data centers |
US8326990B1 (en) | 2005-07-15 | 2012-12-04 | Symantec Operating Corporation | Automated optimal workload balancing during failover in share-nothing database systems |
US20090063892A1 (en) * | 2007-08-27 | 2009-03-05 | International Business Machines Corporation | Propogation by a controller of reservation made by a host for remote storage |
US7904743B2 (en) * | 2007-08-27 | 2011-03-08 | International Business Machines Corporation | Propagation by a controller of reservation made by a host for remote storage |
US20090063487A1 (en) * | 2007-08-29 | 2009-03-05 | International Business Machines Corporation | Securing transfer of ownership of a storage object from an unavailable owner node to another node |
US7778986B2 (en) * | 2007-08-29 | 2010-08-17 | International Business Machines Corporation | Securing transfer of ownership of a storage object from an unavailable owner node to another node |
US8707318B2 (en) * | 2007-10-12 | 2014-04-22 | Microsoft Corporation | Partitioning system including a generic partitioning manager for partitioning resources |
US20090100441A1 (en) * | 2007-10-12 | 2009-04-16 | Microsoft Corporation | Resource assignment system with recovery notification |
US20090100436A1 (en) * | 2007-10-12 | 2009-04-16 | Microsoft Corporation | Partitioning system including a generic partitioning manager for partitioning resources |
US8266634B2 (en) | 2007-10-12 | 2012-09-11 | Microsoft Corporation | Resource assignment system with recovery notification |
US20100086753A1 (en) * | 2008-10-02 | 2010-04-08 | Wade Johnson | Foiled articles and methods of making same |
US8327186B2 (en) * | 2009-03-10 | 2012-12-04 | Netapp, Inc. | Takeover of a failed node of a cluster storage system on a per aggregate basis |
US8145838B1 (en) | 2009-03-10 | 2012-03-27 | Netapp, Inc. | Processing and distributing write logs of nodes of a cluster storage system |
US20100232288A1 (en) * | 2009-03-10 | 2010-09-16 | Coatney Susan M | Takeover of a Failed Node of a Cluster Storage System on a Per Aggregate Basis |
US20120042202A1 (en) * | 2009-04-29 | 2012-02-16 | Thomas Rudolf Wenzel | Global write-log device for managing write logs of nodes of a cluster storage system |
US8261125B2 (en) * | 2009-04-29 | 2012-09-04 | Net App. Inc. | Global write-log device for managing write logs of nodes of a cluster storage system |
US20110178984A1 (en) * | 2010-01-18 | 2011-07-21 | Microsoft Corporation | Replication protocol for database systems |
US20110191299A1 (en) * | 2010-02-01 | 2011-08-04 | Microsoft Corporation | Logical data backup and rollback using incremental capture in a distributed database |
US8825601B2 (en) | 2010-02-01 | 2014-09-02 | Microsoft Corporation | Logical data backup and rollback using incremental capture in a distributed database |
US20120023209A1 (en) * | 2010-07-20 | 2012-01-26 | Robert Adam Fletcher | Method and apparatus for scalable automated cluster control based on service level objectives to support applications requiring continuous availability |
US8650281B1 (en) * | 2012-02-01 | 2014-02-11 | Symantec Corporation | Intelligent arbitration servers for network partition arbitration |
US20140215272A1 (en) * | 2013-01-31 | 2014-07-31 | Alcatel-Lucent Usa Inc. | Preventing extreme coresidency hazards in cloud computing |
US8949655B2 (en) * | 2013-01-31 | 2015-02-03 | Alcatel Lucent | Preventing extreme coresidency hazards in cloud computing |
US20150324416A1 (en) * | 2013-01-31 | 2015-11-12 | Hitachi, Ltd. | Management apparatus and management system |
WO2016010972A1 (en) * | 2014-07-17 | 2016-01-21 | Cohesity, Inc. | Dynamically changing members of a consensus group in a distributed self-healing coordination service |
US10657012B2 (en) | 2014-07-17 | 2020-05-19 | Cohesity, Inc. | Dynamically changing members of a consensus group in a distributed self-healing coordination service |
US9690675B2 (en) | 2014-07-17 | 2017-06-27 | Cohesity, Inc. | Dynamically changing members of a consensus group in a distributed self-healing coordination service |
WO2016108945A1 (en) * | 2014-12-30 | 2016-07-07 | Hewlett Packard Enterprise Development Lp | Cluster arbitration |
US9858011B2 (en) * | 2015-12-16 | 2018-01-02 | International Business Machines Corporation | Repopulating failed replicas through modified consensus recovery |
US20170177232A1 (en) * | 2015-12-16 | 2017-06-22 | International Business Machines Corporation | Repopulating failed replicas through modified consensus recovery |
US11882141B1 (en) | 2017-11-27 | 2024-01-23 | Lacework Inc. | Graph-based query composition for monitoring an environment |
US11677772B1 (en) | 2017-11-27 | 2023-06-13 | Lacework Inc. | Using graph-based models to identify anomalies in a network environment |
US11909752B1 (en) | 2017-11-27 | 2024-02-20 | Lacework, Inc. | Detecting deviations from typical user behavior |
US11689553B1 (en) | 2017-11-27 | 2023-06-27 | Lacework Inc. | User session-based generation of logical graphs and detection of anomalies |
US11792284B1 (en) | 2017-11-27 | 2023-10-17 | Lacework, Inc. | Using data transformations for monitoring a cloud compute environment |
US11637849B1 (en) | 2017-11-27 | 2023-04-25 | Lacework Inc. | Graph-based query composition |
US11831668B1 (en) | 2019-12-23 | 2023-11-28 | Lacework Inc. | Using a logical graph to model activity in a network environment |
US11770464B1 (en) | 2019-12-23 | 2023-09-26 | Lacework Inc. | Monitoring communications in a containerized environment |
US11256759B1 (en) | 2019-12-23 | 2022-02-22 | Lacework Inc. | Hierarchical graph analysis |
US11954130B1 (en) | 2019-12-23 | 2024-04-09 | Lacework Inc. | Alerting based on pod communication-based logical graph |
US20220398259A1 (en) * | 2020-12-23 | 2022-12-15 | Kuyun (Shanghai) Information Technology Co., Ltd. | Online analytical processing precomputation engine optimization method based on object storage and application |
US20220334934A1 (en) * | 2021-04-16 | 2022-10-20 | EMC IP Holding Company LLC | Data storage cluster with quorum service protection |
US11556441B2 (en) * | 2021-04-16 | 2023-01-17 | EMC IP Holding Company LLC | Data storage cluster with quorum service protection |
US20220342769A1 (en) * | 2021-04-21 | 2022-10-27 | EMC IP Holding Company LLC | Application consistent network backup using three phase full quorum |
US11921590B2 (en) * | 2021-04-21 | 2024-03-05 | EMC IP Holding Company LLC | Application consistent network backup using three phase full quorum |
Also Published As
Publication number | Publication date |
---|---|
AU3729500A (en) | 2000-10-16 |
EP1222540A1 (en) | 2002-07-17 |
ATE541259T1 (en) | 2012-01-15 |
EP1222540B1 (en) | 2012-01-11 |
US6401120B1 (en) | 2002-06-04 |
EP1222540A4 (en) | 2002-07-17 |
US20020161889A1 (en) | 2002-10-31 |
WO2000058824A1 (en) | 2000-10-05 |
US6938084B2 (en) | 2005-08-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8850007B2 (en) | Consistent cluster operational data in a server cluster using a quorum of replicas | |
US6938084B2 (en) | Method and system for consistent cluster operational data in a server cluster using a quorum of replicas | |
US6279032B1 (en) | Method and system for quorum resource arbitration in a server cluster | |
US6449734B1 (en) | Method and system for discarding locally committed transactions to ensure consistency in a server cluster | |
EP1171817B1 (en) | Data distribution in a server cluster | |
US6163855A (en) | Method and system for replicated and consistent modifications in a server cluster | |
US6178529B1 (en) | Method and system for resource monitoring of disparate resources in a server cluster | |
US6662219B1 (en) | System for determining at subgroup of nodes relative weight to represent cluster by obtaining exclusive possession of quorum resource | |
US6243825B1 (en) | Method and system for transparently failing over a computer name in a server cluster | |
US10360113B2 (en) | Transaction recovery in a transaction processing computer system employing multiple transaction managers | |
US7739541B1 (en) | System and method for resolving cluster partitions in out-of-band storage virtualization environments | |
US7437386B2 (en) | System and method for a multi-node environment with shared storage | |
US6789213B2 (en) | Controlled take over of services by remaining nodes of clustered computing system | |
US6360331B2 (en) | Method and system for transparently failing over application configuration information in a server cluster | |
US7464378B1 (en) | System and method for allowing multiple sub-clusters to survive a cluster partition | |
KR100368078B1 (en) | Method and apparatus for building and managing multi-clustered computer systems | |
KR100336729B1 (en) | Method and apparatus for managing clustered computer system | |
US7478114B2 (en) | Failure tolerant transaction processing system | |
US20020078312A1 (en) | Support for single-node quorum in a two-node nodeset for a shared disk parallel file system | |
US7120821B1 (en) | Method to revive and reconstitute majority node set clusters | |
KR20050035301A (en) | A data processing system adapted to integrating non-homogeneous processes | |
US8127175B2 (en) | High availability of JCA inflowed transactions | |
US11960478B2 (en) | Database system with transactional commit protocol based on safe conjunction of majorities |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MICROSOFT CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GAMACHE, ROD;MASSA, MICHAEL T.;SHRIVASTAVA, SUNITA;AND OTHERS;REEL/FRAME:018354/0873;SIGNING DATES FROM 20020221 TO 20020226 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034766/0001 Effective date: 20141014 |