Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.


  1. Advanced Patent Search
Publication numberUS20150066833 A1
Publication typeApplication
Application numberUS 14/537,527
Publication date5 Mar 2015
Filing date10 Nov 2014
Priority date26 May 2009
Also published asUS8886804, US20100306371
Publication number14537527, 537527, US 2015/0066833 A1, US 2015/066833 A1, US 20150066833 A1, US 20150066833A1, US 2015066833 A1, US 2015066833A1, US-A1-20150066833, US-A1-2015066833, US2015/0066833A1, US2015/066833A1, US20150066833 A1, US20150066833A1, US2015066833 A1, US2015066833A1
InventorsRoger Frederick Osmond
Original AssigneePi-Coral, Inc.
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Method for making intelligent data placement decisions in a computer network
US 20150066833 A1
A method for making data placement decisions in a computer network uses multiple factors comprising social rules (rules, factors and criteria common to all participating nodes and intended to benefit the community of nodes), as well as rules, factors and criteria driven by individual self-interest of the participating nodes. The method calls for each node to act in a semi-autonomous manner, without the need for a central coordinating node. By considering multiple factors fully, and not eliminating factors by a sequence of True/False decisions, the method may arrive at optimal decisions and may generate a ranked list of node candidates.
Previous page
Next page
What is claimed is:
1. A method for making data placement decisions in a computer network, wherein multiple criteria, rules or factors may be considered, prioritized and weighted using one or more algorithms,
wherein such criteria, rules or factors may comprise shared social rules that govern the behavior of all participating nodes, per-object criteria (also called demand criteria) and criteria that relate to a participating node's self-interest and,
wherein the criteria, rules and factors to consider may be predefined, assigned, or derived from current data and,
wherein algorithms for weighting criteria, rules and factors may be predefined, assigned or derived from current data.
2. The method of claim 1 wherein data placement involves whole objects, composites of whole objects, parts of whole objects, and composites of parts of whole objects.
3. The method of claim 1 wherein a participating node may consider object popularity and replica count, and may also consider factors that reflect its own self-interest such as its current load, the overall density of requests being redirected to it, and its own storage or networking costs.
4. The method of claim 1 wherein a participating node may consider the relative affinity of an object to other objects stored by a participating node,
wherein an object's affinity to other objects includes but is not limited to:
a function of the completeness of the set of objects to which an object belongs or,
a function of the frequency with which requests for an object occur with or near requests for other objects.
5. The method of claim 1 wherein performance levels in the network, in terms of overall throughput, node utilization, latency or other factors may be considered.
6. The method of claim 1 wherein object or data set resilience, robustness, reliability or availability levels may be considered.
7. The method of claim 1 wherein there may be no need for a central coordinating node.
8. The method of claim 1 wherein a participating node asks another participating node to accept responsibility for an object or set of objects.
9. The method of claim 1 wherein a participating node (first node), upon receiving a request for an object or set of objects which the first node does not have, the first node may redirect the request to a peer node on which the object or set of objects is known to be stored, or is likely to be stored and,
wherein such redirection is recorded such that a history of redirection requests, relating to nodes, to objects and to sets of objects, may be used as a factor for subsequent data placement decisions.
10. The method of claim 1 wherein data placement decisions may be triggered by events including but not limited to time-related events, access events, threshold events in the network or on one or more participating nodes.
11. The method of claim 1 wherein all relevant criteria are considered in all calculations, without prematurely eliminating a criterion, rule or factor.
  • [0001]
    This invention claims priority to U.S. Provisional Patent Application No. 61/180,991 entitled “Method for making intelligent data placement decisions in a computer network” filed May 26, 2009.
  • [0002]
    The method described in the present application is based on and incorporates in its entirety co-pending U.S. application Ser. No. 12/782,495 entitled “Method for making optimal selections based on multiple objective and subjective criteria” [OSMOND01], by Roger F. Osmond, filed on May 18, 2010.
  • [0003]
    This invention relates generally to computer software. Aspects of this invention also relate more specifically to network software, data storage and replication.
  • [0004]
    Data storage systems are responsible for the safety and integrity of the data they host. One common method used to protect data from loss is data replication. Data replication makes additional copies of data, often on different storage nodes. When there are multiple copies of a single data item, the probability of loss of all of the copies is much lower than for data on a single node.
  • [0005]
    An added benefit of replication, and often a major motivation for it, is improved overall read performance. By having copies of data on multiple nodes, read activity in the network is less likely to concentrate on one or very few nodes, and will likely be more balanced across the nodes. In larger scale networks spanning different regions of geography, connectivity, organization and administration, having one or more copies within a region can improve responsiveness for applications executing in that region.
  • [0006]
    It would be advantageous to have a method by which data replicas would be placed intelligently amongst the nodes of a network, without excessive management overhead or complexity, and with reasonable efficiency.
  • [0007]
    In accordance with the present invention, a method for making data placement decisions in a computer network is provided which uses multiple factors comprising social rules and the self-interest of the participating nodes. The method calls for each node to act in a semi-autonomous manner, without the need for a central coordinating node.
  • [0000]
      • Table 1 depicts an example candidate/criterion matrix of nodes.
  • [0009]
    The present invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings in which:
  • [0010]
    FIG. 1 depicts a logic flow representing the present invention's method for data placement (replication) decision making,
  • [0011]
    FIG. 2 depicts a simplified request processing flow, and
  • [0012]
    FIG. 3 depicts the effects of different weighting algorithms on prioritized criteria.
  • [0013]
    The method of the present invention provides to each participating node a set of rules of conduct, “social rules”. Each node acts according to these social rules when making decisions regarding data placement. Each node also has to consider demand-side criteria and self-interest criteria. FIG. 1 depicts a (simplified) flow of logic that a storage node executes when making data placement decisions. Item 101 denotes a step in the logic at which the node decides whether or not a replica should be created for that object. At this step, a node considers demand criteria (Item 102) like object popularity and replica count. It also considers factors that reflect its own self-interest (Item 103) such as its current load, the overall density of requests being redirected to it, and its own storage costs.
  • [0014]
    If warranted, then the node continues (Item 105) with the replication process by selecting the best target node or nodes from a candidate list (peer nodes). The selection process considers multiple criteria including those related to self-interest (Item 106) and those imposed by social rules (Item 107).
  • [0015]
    The preferred embodiment treats all replicas as peers, though a scheme by which there are primary and subordinate replicas is also possible. The preferred embodiment implements a redirection mechanism in which, upon a node receiving a request for an object not present locally on that node, the receiving node redirects the request to a peer node on which the requested object is present, as illustrated in FIG. 2.
  • [0016]
    Self-interest factors include Storage Cost, Load, Redirection Density and Object-Node Affinity. This is a representative list only, and other factors are likely. Storage Cost relates to the actual cost of the storage devices and of storing data on them (including equipment purchase/lease, and operational costs). For example, a SAS (Serial-Attached SCSI) or SCSI (Small Computer System Interface) drive might be more costly than a SATA (Serial Advanced Technology Attachment) drive of similar capacity. Load relates to the current, recent and historical load on the storage node making the decision. Redirection Density relates to the recent and historical frequency of redirection requests for the object being considered.
  • [0017]
    Object-Node Affinity, as it pertains to the present invention, is a reflection of the completeness of the object's context on the decision-making node. In a simple case, an object's context would include the object's container (e.g. a folder or directory) and with it, the other objects that could be in that container. The degree of completeness of a context (e.g. the extent to which that container is already populated) can be considered as a factor indicating the likelihood that other data (e.g. other objects from the same container) would be requested.
  • [0018]
    For example, a storage system might arrange data objects into containers. A node might hold a significant portion of the objects in a container. A second node might have a high incidence of requests for objects in that container, forwarding or redirecting those requests to the first node. Based on a combination of social rules and self-interest, the first node might, over time, create on the second node replicas of a number of objects in that object's container. At some point (e.g. at predetermined intervals, or in response to a request), the first node might assess the extent to which the second node resembles the first, with respect to the completeness of the container. The first node then, perhaps to reduce the anticipated density of future requests for objects in that container, may decide to create on the second node replicas of the remaining objects in that container, so long as doing so does not violate any social rules. In this manner, the first node is acting in its own self-interest to create a replica on the second node.
  • [0019]
    Social rule-based factors include Replica Min/Max, Object Popularity, and Movement Cost. This is a representative list only, and other factors are likely, including object criticality, retention policies and object age.
  • [0020]
    Replica Min/Max might be a global (i.e. all nodes, all spaces and all objects) rule, or might be qualified. It denotes the working minimum and maximum replica counts either for objects in general, or for the object being considered for replication. When the current count is at or below the minimum, then a new replica needs to be created. One of the nodes on which a replica resides would initiate creation of a new replica. When the current count is at or above the maximum, additional replicas are not required. Typically the minimum relates to service levels and the maximum is driven by cost considerations, but object popularity can factor into the equation as well.
  • [0021]
    Object popularity is the measured and/or predicted popularity of a given object. For example, a much anticipated new movie title might have a very high initial predicted popularity and so would warrant a higher number of replicas. Localized popularity might vary and the popularity of an object can change relatively quickly. Object popularity can also be used, in conjunction with storage cost, to reduce the number of replicas or to move objects (replicate then delete) to servers offering less costly storage (or both). The method of this invention then provides the benefits of a content distribution network (CDN) and hierarchical storage management (HSM), into a more comprehensive value-driven storage model.
  • [0022]
    Movement cost, while governed by social rules, relates also to self-interest. Movement cost is the relative cost of copying or moving a given object from one node to another. The cost is calculated primarily on object size relative to network bandwidth. From the social rules perspective, each storage node must behave in a way that is socially responsible, for example not burdening the network disproportionately. In other words, unless there is a compelling reason to do so, a node should not create a replica if doing so imposes an undue burden on the network (a shared resource).
  • [0023]
    Object-peer affinity is a measure of an object's affinity with a node that is a peer to the current storage node (i.e. a node that closely resembles the current node with respect to a given object's context), and is related to the object-node affinity factor for self-interest analysis. Simply stated, object-peer affinity is analogous to object-node affinity, except that object-peer affinity relates the object to a peer of the decision making node, instead of relating to the decision making node itself.
  • [0024]
    For illustration, an object might belong to a collection of objects, the overwhelming majority of which have replicas on a peer node. The node performing the analysis (the first node) might also have a strong affinity with the object in question and have a relatively low redirection density from the second node. As such, the first node would not be inclined to create a replica for that object based solely on the first node's self-interest. The object-peer affinity factor would encourage (but not mandate) the first node to create the replica.
  • [0025]
    The decision to create a replica, and on which node, is a potentially complex task. Taken singly, the factors that must be considered might result in several different decisions. The method of the present invention considers all relevant factors in its data placement decisions. It does this by exploiting and adapting a decision-making method described in [OSMOND01].
  • [0026]
    The method of the present invention assembles a matrix of decision factors (criteria) and candidate nodes to which to apply the method adapted from [OSMOND01].
  • [0000]
    TABLE 1
    Example Node/Factor Matrix
    Redirection Total Contiguous Object
    Storage Cost Density Capacity Capacity Target Load Popularity Raw Totals
    Node 1 5 5 3 3 3 5 24
    Node 2 2 5 5 5 1 3 21
    Node 3 1 1 2 4 5 3 16
    Node 4 5 1 1 1 3 3 14
    Node 5 2 2 5 5 4 3 21
  • [0027]
    Table 1 shows an example decision matrix with 5 candidate nodes and 6 decision criteria (in reality there might be several more of each). Each criterion has been given a relative value through the normalization process of the decision making method, where a value of 5 (for 5 candidates) is the highest and 1 is the lowest. It is important to note that factors such as the decision-making node's load and the object's affinity with the current node are absent from that table, as they are the same regardless of the candidate node, and would be considered in a separate step that decides whether or not to create a replica at all (See FIG. 1, Item 101).
  • [0028]
    The raw totals in Table 1 seem to indicate that Node 1 has the highest points and is therefore the preferred target. According to the method in [OSMOND01], the criteria would be sorted by priority and then weighted to produce a more meaningful decision (the criteria in Table 1 are not sorted in any particular way). The raw values in Table 1 imply that Node 1 is the preferred target for a replica. Table 2 lists the raw values of Table 1 with various weightings applied. In each case, Node 1 is the top candidate. The weightings apply to the prioritized criteria to reflect more accurately the relative importance in decision making, as described in [OSMOND01]. Raw totals are un-weighted and therefore do not reflect criteria priority. Any number of algorithms could be used to apply priority weighting, and the ones used in Table 2 are for illustration.
  • [0000]
    TABLE 2
    Example Weighted Node/Factor Matrix
    Raw Linear Log Exponential
    Node 1 24 1830.0 2147.4 1477.0
    Node 2 21 1610.0 1890.6 1281.0
    Node 3 16 1080.0 1365.7 760.0
    Node 4 14 1070.0 1250.8 877.0
    Node 5 21 1520.0 1845.6 1146.0
  • [0029]
    Changing the priority of the criteria has an interesting effect on the outcome of the decision making process.
  • [0000]
    TABLE 3
    Alternate Criteria Priorities
    Original Alternative
    Storage Cost Target Load
    Redirection Density Total Capacity
    Total Capacity Contiguous Capacity
    Contiguous Capacity Redirection Density
    Target Load Object Popularity
    Object Popularity Storage Cost
  • [0030]
    By changing the priority from that in Table 1 to Target Load, Total Capacity, Contiguous Capacity, Redirection Density, Object Popularity and then Storage Cost (Table 3), the preferred node changes for the Exponential weighting option, as shown in Table 4, from Node 1 to Node 5, and rather decisively.
  • [0000]
    TABLE 4
    Example Weighted Node/Factor Matrix with Alternate Priority
    Raw Linear Log Exponential
    Node 1 24 1710.0 2090.7 1285.0
    Node 2 21 1580.0 1878.0 1228.0
    Node 3 16 1300.0 1473.9 1100.0
    Node 4 14 970.0 1199.9 727.0
    Node 5 21 1670.0 1918.2 1381.0
  • [0031]
    The Exponential weighting option in the example gives significantly greater importance to the higher priority criteria than to the lower priority criteria. This might be desirable when a single factor is especially compelling. In other contexts, a less severe weighting algorithm might be appropriate. FIG. 3 depicts the effects of different weighting algorithms on prioritized criteria in the example, with Graph 3 in FIG. 3 depicting the effect of the Exponential weighting on the criteria priority reflected in Table 4.
  • [0032]
    The rules that govern criteria priority and weighting are defined for each node in the network and are typically the same for each. In some cases, some nodes might be special in some way (e.g. nodes bridging multiple subnets or nodes with different service level agreements), and therefore might have different rules than the other nodes in that network.
  • [0033]
    By being able to make well-informed data placement decisions, balancing demand-side and supply-side factors, storage nodes are able to achieve higher performance, higher service levels, and higher capacity utilization. This capability would be of significant benefit, but not be limited to, content-distribution networks, video-on-demand services and Internet data services in general.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US6256675 *6 May 19983 Jul 2001At&T Corp.System and method for allocating requests for objects and managing replicas of objects on a network
US6463454 *17 Jun 19998 Oct 2002International Business Machines CorporationSystem and method for integrated load distribution and resource management on internet environment
US6466980 *17 Jun 199915 Oct 2002International Business Machines CorporationSystem and method for capacity shaping in an internet environment
US8087025 *29 Jun 200527 Dec 2011Hewlett-Packard Development Company, L.P.Workload placement among resource-on-demand systems
US20050216428 *24 Mar 200429 Sep 2005Hitachi, Ltd.Distributed data management system
US20050283487 *21 Jun 200422 Dec 2005Magnus KarlssonMethod of determining lower bound for replication cost
US20070156842 *8 Mar 20065 Jul 2007Vermeulen Allan HDistributed storage system with web services client interface
US20080313328 *20 Aug 200818 Dec 2008Intellectual Ventures Holding 40 LlcMethod and system for background replication of data objects
US20100274762 *24 Apr 200928 Oct 2010Microsoft CorporationDynamic placement of replica data
US20100325476 *12 Aug 201023 Dec 2010Microsoft CorporationSystem and method for a distributed object store
US20120016907 *13 Jul 201119 Jan 2012Tatsuro IkedaMethod and apparatus for controlling replication processing of object
US20130054518 *30 Aug 201128 Feb 2013International Business Machines CorporationApplying replication rules to determine whether to replicate objects
U.S. Classification706/47
International ClassificationH04L12/24, G06N5/02
Cooperative ClassificationH04L67/1095, H04L41/147, G06N5/027, H04L67/1023, G06F17/30575, H04N21/231, H04L67/101, H04N21/00, G06F11/1492, H04L67/1008, H04L29/0854
Legal Events
12 Dec 2014ASAssignment
Effective date: 20140827