US20030217211A1 - Controller communications over an always-on controller interconnect - Google Patents
Controller communications over an always-on controller interconnect Download PDFInfo
- Publication number
- US20030217211A1 US20030217211A1 US10/146,546 US14654602A US2003217211A1 US 20030217211 A1 US20030217211 A1 US 20030217211A1 US 14654602 A US14654602 A US 14654602A US 2003217211 A1 US2003217211 A1 US 2003217211A1
- Authority
- US
- United States
- Prior art keywords
- controller
- loop
- data
- level
- pair
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2089—Redundant storage control functionality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2002—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where interconnections or communication control functionality are redundant
- G06F11/2007—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where interconnections or communication control functionality are redundant using redundant communication media
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2002—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where interconnections or communication control functionality are redundant
- G06F11/2007—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where interconnections or communication control functionality are redundant using redundant communication media
- G06F11/201—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where interconnections or communication control functionality are redundant using redundant communication media between storage system components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2089—Redundant storage control functionality
- G06F11/2092—Techniques of failing over between control units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
Definitions
- the present disclosure relates to disk arrays, and more particularly, to a controller interconnect structure within multi-controller disk arrays that permits continued communication between controllers under various failure scenarios.
- Modem mass storage systems continue to provide increasing storage capacities to meet user demands from host computer system applications.
- a growing reliance on large capacity mass storage has fueled a corresponding demand for enhanced reliability of such storage systems.
- One popular solution to the demands for increased storage capacity and reliability is the use of multiple smaller storage modules configured in geometries that permit redundancy of stored data to assure data integrity in case of various failures.
- RAID (redundant array of independent disks) disk arrays are an example of a fault tolerant, mass storage technology that has developed in response to the ever-increasing demands for greater storage capacity and reliability.
- RAID disk arrays supply host computer systems with large amounts of storage capacity in addition to providing redundancy of stored data to assure data integrity in case of various failures.
- Such disk arrays therefore typically include redundant components such as controllers and power supplies, as well as hot-swap capabilities for various subsystem modules (i.e., an ability to change-out modules without powering down the system).
- RAID arrays commonly have two controllers that manage the array and perform mirrored memory operations for data redundancy.
- the controllers make the array appear to the host computer as a single, highly reliable, high capacity disk drive.
- Both controllers have independent access to all data cache information, all input/output (I/O) state information, and all system state information so that a failure of one of the controllers does not prevent the remaining working controller from accessing all the necessary information to take over sole operation of the array.
- Significant bandwidth is required on controller interconnect buses to allow the controllers to transfer the necessary information for processing host I/O requests and performing mirrored memory operations.
- controller pairs can be added to the arrays to increase their computing resources and maintain or improve system performance.
- the number of controller pairs increases, the amount of data flowing between controllers over the controller interconnect buses increases dramatically.
- the controller interconnect not only carries mirrored data traffic and inter-processor communications between controllers in pair “B”, but it also carries pair-to-pair traffic between the two controller pairs “A” and “B”.
- the interconnect must carry the traffic from the controller board in pair “A” that received the host data to the controller board in pair “B” that is the destination of the host data.
- the interconnect must carry the mirror traffic between the two controller boards that form controller pair “B”. Therefore, an increase in the number of controller pairs within a disk array can contribute to performance bottlenecks due to bandwidth limitations of the controller interconnect buses.
- controller interconnects Another consideration regarding controller interconnects is emerging technologies that allow for wider interfaces between disk arrays and host systems. As higher performance host computer connections are developed for connecting RAID storage arrays to host computer systems, controller interconnect buses experience a corresponding increase in the amount of data flowing between controllers within an array. Again, bandwidth limitations on controller interconnect buses within the array can result in performance bottlenecks.
- Another problem that results from adding more controllers to a disk array is that more and more data travels to remote controllers rather than a local controller in the mirrored controller pair where the data is received.
- all the host computer disk traffic is destined for the local mirrored cache because there is only one mirrored cache.
- the percentage of data flowing to the local mirrored cache drops to 50%.
- Half the traffic stays with the local cache memory while the other half is destined for the remote pair's cache memory.
- With 16 pairs of controllers only about 7% of the traffic is local.
- the characteristics of the controller interconnect changes dramatically with the clustering of controllers in the disk array.
- a controller interconnect structure permits low latency/high bandwidth communications through mirror buses that couple controllers together as mirrored controller pairs within a RAID disk array having a plurality of mirrored (i.e. clustered) controller pairs.
- the interconnect structure also forms a controller loop that couples controllers together through loop buses.
- the controller loop provides an automatic fail-over function that enables continued communications between controller pairs in the event that a failure occurs within the controller loop.
- a disk array includes at least two pairs of controllers. Each controller pair has a first and second controller that perform mirrored memory operations through a mirror bus that carries mirror data traffic between the two mirrored controllers in the controller pair.
- a controller loop is formed by the interconnection of all the controllers through a plurality of loop buses. Each controller is coupled to two logically adjacent controllers through a loop bus such that a continuous loop of controllers is formed. The controller loop permits data and control information to travel in both directions along the loop between the two or more controller pairs. Routing logic in each controller controls the flow of data in the loop such that data packets are normally routed to the nearest mirrored controller associated with array addresses in the data packet headings.
- a failure in the controller loop causes the loop to fail into a single string of controllers. Although the controller loop has failed, the controller string is capable of providing continued communication between all controller pairs. Hardware circuitry on each controller provides an automatic fail-over function that tolerates failures in the controller loop by detecting a failure and rerouting data in a different direction to avoid the failure. Therefore, data initially traveling in one direction through the loop will be rerouted or “bounced” in the opposite direction when a loop failure is encountered. The controller string then carries the data to its destination controller pair.
- Another embodiment includes the controllers configured as in the prior embodiment, coupled together through two back plane interconnect boards.
- One half of each mirror bus and one half of each loop bus runs through each of the two back planes.
- Both halves of each bus can work in unison under normal operation or one half of each bus is able to take over all the data traffic of both halves in the event of a failure condition.
- the dual back plane configuration permits on-line repair of either back plane.
- Hardware circuitry on controller boards automatically detects failed links between boards. A detected failure on any bus automatically fails the bus over to using the operational half of the bus.
- either one of the two back planes can be removed and repaired while data continues flowing between controllers over the operational half of each bus that runs through the remaining back plane.
- a third embodiment includes two or more sets of controllers generally configured as in the prior embodiments and logically coupled into levels.
- the embodiment allows the size of a disk array system to be scaled up significantly by expanding the number of controllers through additional controller loops.
- Each loop of controllers is configured as a level of controllers stacked upon another level of controllers.
- Each additional controller loop has the same properties as in the previously described embodiments where broken or failed links do not disable the transfer of data through the system.
- controllers include programmable routing registers that contain routing information to control the direction of data flow along a controller loop.
- the routing registers permit a matched data flow along loop bus segments so that no single loop bus segment is over burdened.
- the controller's routing logic is configured to access the routing information from the programmable routing register to determine the direction in which to send the I/O command data.
- routing logic is configured to reprogram routing registers when a failure occurs in a controller loop so that data flow between controllers is more efficient.
- Hardware circuitry automatically detects a failure and reroutes data to avoid the failure. However, continually sending data in one direction and then rerouting it by a “hardware” reroute is not the most efficient use of the interconnect structure's capacity. Therefore, when hardware detects a failure and reroutes data, it also notifies the routing logic of the failure so routing registers will be reprogrammed to provide modified data routes that avoid the failure without traversing the less efficient hardware reroute. Data is thus initially routed in a direction that avoids the failure.
- the hardware detection circuitry can also be configured to reprogram routing registers.
- FIG. 1 illustrates a system environment that is suitable for implementing an arrayed storage device having an always-on controller interconnect structure.
- FIG. 2 is a block diagram illustrating in greater detail, a particular embodiment of the system environment of FIG. 1 including a host computer device and an arrayed storage device implemented as a RAID disk array having an always-on controller interconnect structure.
- FIG. 3 is a block diagram illustrating in greater detail, a controller pair such as the controller pair illustrated in the block diagram of FIG. 2.
- FIG. 4 shows a logical representation of a controller interconnect structure such as might be implemented in the RAID disk array of FIG. 2.
- FIG. 5 illustrates the controller interconnect structure of FIG. 4 under a particular failure scenario.
- FIG. 6 illustrates a controller interconnect structure such as that of FIG. 4 in a redundant back plane configuration.
- FIG. 7 illustrates the controller interconnect structure of FIG. 6 under a particular repair scenario.
- FIG. 8 shows a logical representation of another embodiment of a controller interconnect structure having an additional controller interconnect level such as might be implemented in the RAID disk array of FIG. 2.
- FIG. 9 is a flow diagram illustrating an example of a general method of performing controller communications over an always-on controller interconnect structure.
- a controller interconnect structure within a RAID disk array enables continuous low latency/high bandwidth communications between a plurality of controller pairs within the array.
- Mirror buses carry high speed mirror traffic between mirrored controllers performing mirrored memory operations.
- Loop buses carry inter-processor communications and other traffic between controller pairs coupled together in a controller loop.
- Benefits of the interconnect structure include an ability to support continued controller communications and online disk array operations under various failure and repair conditions that might otherwise render a disk array inoperable.
- the controller interconnect structure provides for easy expansion of the number of controllers within disk arrays as arrays continue to be scaled up in size to meet increasing storage demands from user host systems.
- FIG. 1 illustrates a system environment 100 suitable for implementing an always-on controller interconnect structure.
- the system 100 includes arrayed storage device (e.g. a RAID storage array) 102 operatively coupled to host device(s) 104 through network 106 .
- Storage device 102 typically provides for multiple redundant network connections 106 .
- Network connection 106 can include, for example, a LAN (local area network), a WAN (wide area network), an intranet, the Internet, a fiber optic cable link, a wireless link, a direct connection, or any other suitable communication link.
- Host device(s) 104 can be implemented as a variety of general purpose computing devices including, for example, a personal computer (PC), a laptop computer, a server, a Web server, and other devices configured to communicate with arrayed storage device 102 .
- arrayed storage device 102 is not limited in this regard. Accordingly, this disclosure is applicable to other configurations of arrayed storage components as currently exist or as might exist in the future that include different array architectures intended to offer high-performance, fault-tolerant mass storage similar to that provided by currently available RAID systems. Therefore, arrayed storage device 102 more generally refers to a plurality of storage components/devices operatively coupled in an array for the general purpose of increasing storage performance. Storage performance goals typically include storage capacity, low cost per stored megabyte, high input/output performance, and high data availability through redundancy and fault tolerance. Storage components/devices operatively coupled within arrayed storage devices 102 may include devices such as magnetic disk drives, tape drives, optical read/write disk drives, solid state disks and the like. Such storage components are generally well known in the art of data storage technology.
- FIGS. 2 and 3 are block diagrams illustrating a particular embodiment of a host computer device 104 and an arrayed storage device 102 as might be implemented in the system environment 100 of FIG. 1.
- the arrayed storage device 102 of FIG. 1 is embodied in FIG. 2 as a RAID storage array 102 having a plurality of clustered controller pairs 208 .
- Host device 104 is embodied generally as a computer such as a personal computer (PC), a laptop computer, a server, a Web server, or other computer device configured to communicate with RAID storage array 102 .
- PC personal computer
- server a server
- Web server or other computer device configured to communicate with RAID storage array 102 .
- Host device 104 typically includes a processor 200 , a volatile memory 202 (i.e., RAM), and a nonvolatile memory 204 (e.g., ROM, hard disk, floppy disk, CD-ROM, etc.).
- Nonvolatile memory 204 generally provides storage of computer readable instructions, data structures, program modules and other data for host device 104 .
- Host device 104 may implement various application programs 206 stored in memory 204 and executed on processor 200 that create or otherwise access data to be transferred via network connection 106 to RAID storage array 102 for storage and subsequent retrieval.
- Such applications 206 might include software programs implementing, for example, word processors, databases, spread sheets, browsers, multimedia players, illustrators, computer-aided design tools and the like.
- host device 104 provides a regular flow of data I/O requests to be serviced by RAID storage array 102 .
- RAID storage array 102 is generally designed to provide continuous data storage and data retrieval for computer devices such as host device(s) 104 , and to do so under various fault conditions that may occur.
- RAID array 102 typically includes redundant subsystems such as controller pairs 208 and power and cooling subsystems 210 that permit continued access to the RAID array 102 even during a failure of one of the subsystems.
- RAID array 102 typically provides hot-swapping capabilities for array components (i.e. the ability to remove and replace components while the array 102 remains online) such as the controllers in controller pairs 208 , the power/cooling subsystems 210 , and the disk drives 214 in the array of disks 212 .
- Each controller pair on RAID array 102 includes a first controller (e.g., CTLR A 1 ) and a second controller (e.g., CTLR A 2 ).
- the two controllers in each controller pair 208 mirror each other and are generally configured to redundantly store and access data on disk drives 214 .
- controllers A 1 and A 2 perform tasks such as mapping host data to disk drives, performing RAID calculations, mirroring data between redundant controller boards, attaching validation tags to data before saving the data to disk drives 214 and checking the tags to ensure data from a disk drive 214 is correct before sending the data back to a host device 104 .
- Controllers in each controller pair 208 also tolerate faults such as disk drive 214 failures by recreating data that may be lost during such failures.
- FIG. 3 is a block diagram illustrating an example of a controller Al from a controller pair 208 ( 1 ) in more detail.
- Controller A 2 from controller pair 208 ( 1 ) is represented in FIG. 3 but is not specifically detailed because it is configured the same as controller Al.
- each controller in a controller pair 208 on RAID array 102 typically includes I/O processor(s) such as FC (fiber channel) I/O processor(s) 216 , main processor(s) 218 , nonvolatile (NV) RAM 220 , memory 222 (e.g., ROM, RAM), and one or more ASICs (application specific integrated circuits) such as memory control ASIC 224 .
- I/O processor(s) such as FC (fiber channel) I/O processor(s) 216 , main processor(s) 218 , nonvolatile (NV) RAM 220 , memory 222 (e.g., ROM, RAM), and one or more ASICs (application specific integrated circuits) such as memory control ASIC 224
- NV RAM 220 is typically supported by a battery backup (not shown) that preserves data in NV RAM 220 in the event power is lost to controller(s) 208 .
- Memory 222 generally provides storage of computer readable instructions, data structures, program modules and other data for RAID storage array 102 . Accordingly, nonvolatile memory 222 includes firmware 226 which is generally configured to execute on processor(s) 218 and support normal disk array 102 operations. Firmware 226 is also typically configured to handle various fault scenarios that may arise in RAID array 102 .
- routing logic 228 and routing register(s) 230 are configured to route data between various controller pairs 208 via a controller interconnect structure. Also discussed more fully below is a hardware detection and rerouting circuit 232 that is generally configured to detect controller interconnect failures and reroute data in order to circumvent such failures.
- FC I/O processor(s) 216 on controllers receives data and commands from host device 104 via network connection 106 .
- FC I/O processor(s) 216 communicates with main processor(s) 218 through standard protocols and interrupt procedures to transfer data and commands to redundant controllers (e.g., controller A 2 of FIG. 3) and generally move data between NV RAM 220 and various disk drives 214 to ensure that data is stored redundantly.
- Memory control ASIC 224 generally controls data storage and retrieval, data manipulation, redundancy management, and the like through communications between mirrored controllers such as controllers Al and A 2 of FIG. 3, for example.
- Memory controller ASIC 224 handles mirroring of data between controllers, tagging of data sectors being striped to disks 214 in the array of disks 212 and RAID calculations to write parity information across the disk drives 214 , as well as data reconstruction in the event of a disk drive failure. Data striping and parity checking are well-known to those skilled in the art.
- Memory control ASIC 224 also typically includes internal buffers (not shown) that facilitate testing of memory 222 to ensure that all regions of mirrored memory (e.g., between mirrored controllers A 1 and A 2 ) are compared to be identical and checked for ECC (error checking and correction) errors on a regular basis. Memory control ASIC 224 notifies processor 218 of these and other errors it detects.
- Firmware 226 is configured to manage errors detected by memory control ASIC 224 in a tolerant manner which may include, for example, preventing the corruption of array 102 data or working around a detected error/fault through a redundant subsystem to prevent the RAID array 102 from crashing.
- FIG. 4 illustrates an example of an always-on controller interconnect structure that is suitable for implementation in the RAID storage array 102 of FIGS. 1 and 2.
- FIG. 4 includes a plurality of controller pairs 208 interconnected through a network of mirror buses 400 (represented by solid-lined arrows) and loop buses 402 (represented by dashed-lined arrows).
- Each controller pair 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), and 208 ( 4 ) includes a first and second controller operatively coupled through a first interconnect, or mirror bus 400 .
- controller A 1 is coupled to controller A 2 through a mirror bus 400
- controller B 1 is coupled to controller B 2 through another mirror bus 400 , and so on.
- the mirror buses 400 carry mirror traffic between the two controllers in each of the mirrored controller pairs 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), and 208 ( 4 ).
- the mirror buses 400 between each controller pair 208 are therefore capable of providing low latency/high bandwidth transfers as host data and RAID maps are stored and accessed in mirrored memory (i.e. NV RAM 220 ).
- Each controller in the controller interconnect structure of FIG. 4 is additionally coupled to two other logically adjacent controllers through a second interconnect, called loop buses 402 .
- the two logically adjacent controllers coupled to a particular controller through loop buses 402 do not include that particular controller's mirrored controller, which is already coupled through a mirror bus 400 .
- controller B 1 for example, is coupled to logically adjacent controllers A 1 and C 1 through loop buses 402 .
- the controller interconnect structure includes two points where loop buses 402 cross over between first controllers from the controller pairs 208 to second controllers from the controller pairs 208 .
- the cross over forms a connection between a row of first controllers (i.e., A 1 , B 1 , C 1 , and D 1 ) and a row of second controllers (i.e., A 2 , B 2 , C 2 and D 2 ), which in turn forms a continuous loop of controllers.
- a loop bus 402 crosses over to couple first controller A 1 to logically adjacent second controller D 2 .
- a loop bus 402 crosses over to couple first controller D 1 to logically adjacent second controller A 2 .
- each controller forms part of a continuous controller loop by virtue of being coupled to two logically adjacent, but non-mirrored, controllers.
- mirror buses 400 typically carry mirror traffic between two controllers within a mirrored controller pair (e.g., 208 ( 1 ))
- the loop buses 402 carry traffic between the various controller pairs.
- Pair-to-pair traffic includes data received at one controller (e.g., controller A 1 of controller pair 208 ( 1 )) that is destined for another pair of mirrored controllers (e.g., controller pair 208 ( 3 )) in addition to all IPC (inter-processor communication) traffic.
- Pair-to-pair traffic flows in both directions around the controller loop.
- routing logic 228 and routing register(s) 230 are configured to route data between various controller pairs 208 via the controller interconnect structure.
- the routing logic 228 routes data along the continuous controller loop (see FIG. 4) so that data arrives at its destination via the quickest route and, so that each segment of the controller loop is used efficiently.
- the routing logic 228 determines the best direction to send data along the controller loop based on information/instructions from routing register(s) 230 and on the array's mapping of host addresses to array addresses.
- a controller that receives the data assigns a header or data packet heading that identifies which controller pair 208 is the proper destination for the data.
- the routing logic 228 uses routing register 230 instructions associated with the data header to send the data in a direction along the controller loop which typically, but not necessarily, takes the data to the nearest mirrored controller of the destination controller pair. For example, data received from a host 104 by controller A 1 208 ( 1 ) that is destined for controller pair B 208 ( 2 ) will be routed to the right, over the single loop bus 402 segment between controllers A 1 and B 1 . Thus, the data is routed to the nearest mirrored controller B 1 of the destination controller pair B 208 ( 2 ).
- Routing register(s) 230 are programmable registers located in the routing logic 228 that provide the routing logic 228 with information on which direction to send data destined for a controller pair 208 . Routing register(s) 230 are initially programmed, for example, by processor 218 to contain information that the routing logic 228 uses to determine which direction to route data over the controller loop (see FIG. 4).
- the nearest mirrored controller of a destination controller pair 208 may be equidistant from the controller sending the data.
- data received from a host device 104 by controller A 1 208 ( 1 ) that is destined for controller pair C 208 ( 3 ) is an equal distance away from the nearest mirrored controller of destination controller pair C 208 ( 3 ). That is, it is no closer for the routing logic 228 on controller A 1 to send the data to controller C 1 208 ( 3 ) by way of controller B 1 208 ( 2 ) than it is to send the data to controller C 2 208 ( 3 ) by way of controller D 2 208 ( 4 ).
- routing registers 230 permit an evenly matched flow of data over all loop bus 402 segments of the controller loop.
- routing registers 230 may tell routing logic 228 on each controller to send such equidistant data in the same direction around the controller loop, such as to the right, or clockwise.
- controller B 1 208 ( 2 ) when data at controller B 1 208 ( 2 ) is destined for controller pair D 208 ( 4 ), it is sent to controller D 1 208 ( 4 ) by way of controller C 1 208 ( 3 ), instead of being sent to controller D 2 208 ( 4 ) by way of controller A 1 208 ( 1 ).
- controller A 1 208 ( 1 ) the loop bus 402 segment between controller A 1 208 ( 1 ) and controller B 1 208 ( 2 ) does not get overburdened by excess traffic.
- workloads from host device(s) 104 are spread evenly among the various controllers, the flow of data over each segment of the controller loop is evenly matched.
- FIG. 5 illustrates the controller interconnect structure of FIG. 4 discussed above and includes an example of a failure 500 in the controller loop at a particular point in the loop bus 402 . It is apparent from FIG. 5 that a break in the controller loop interconnect structure causes the continuous controller loop to fail into a controller string. Thus, any failure 500 in the controller loop will mark the end points of the controller string.
- the endpoints of the controller string of FIG. 5 are at controller C 1 208 ( 3 ) and controller B 1 208 ( 2 ).
- the detection/rerouting hardware circuits 232 on controllers C 1 208 ( 3 ) and B 1 208 ( 2 ) automatically detect the loop failure 500 and “bounce” data that encounters the failure 500 back in the opposite direction. For example, if data is traveling from controller A 1 208 ( 1 ) through controller B 1 208 ( 2 ) to controller C 1 208 ( 3 ), the detection/rerouting hardware circuit 232 on controller B 1 208 ( 2 ) will “bounce” the data back over the controller string so it arrives at controller C 1 208 ( 3 ) by way of controller D 1 208 ( 4 ).
- the hardware circuits 232 can provide notification to processor(s) 218 of the failure so that the processor(s) 218 can reprogram the routing registers 230 on the controllers. This enables the routing logic 228 to avoid the failure 500 when it initially routes data over the controller interconnect structure. Reprogramming the routing registers 230 in this manner makes more efficient use of the controller interconnect under a failure condition.
- the hardware circuits 232 may themselves modify the routing information in the routing registers 230 under such failure conditions.
- mirror buses 400 which typically carry mirror traffic between two controllers within a mirrored controller pair (e.g., 208 ( 1 )), can also be used to carry “loop traffic”.
- “loop traffic” data may be routed over a mirror bus 400 between a mirrored controller pair in order to avoid the non-present controllers while still maintaining a full controller loop interconnection. Therefore, the controller loop may be formed using both loop buses 402 and mirror buses 400 .
- hardware circuits 232 would provide some low-level physical presence information to the routing logic 228 that will change the way traffic is routed through the controller loop.
- FIG. 6 illustrates another embodiment of the always-on controller interconnect structure suitable for implementation in the RAID storage array 102 of FIGS. 1 and 2.
- the interconnect structure of FIG. 6 is configured like the FIG. 4 interconnect structure described above.
- mirror buses 400 carry mirror traffic between the two controllers in each of the mirrored controller pairs 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), and 208 ( 4 ), and each controller forms part of a continuous controller loop by virtue of being coupled, by interconnect 402 , to two logically adjacent, but non-mirrored controllers.
- the FIG. 6 embodiment includes a dual back plane configuration that allows for the on-line repair of a failed back plane while the remaining back plane continues to provide a fully functioning interconnect structure between all controllers in the RAID storage array 102 .
- each controller from controller pairs 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), and 208 ( 4 ) is coupled to or plugged into two separate interconnects.
- the interconnects are embodied as back planes 600 and 602 .
- Back plane #1 600 and back plane #2 602 both carry one half of each bus that is shown in the controller interconnect structure of FIG. 4.
- each half of each bus i.e., mirror buses 400 and loop buses 402
- the loop bus 402 that carries pair-to-pair traffic from controller A 1 208 ( 1 ) to controller D 1 208 ( 4 ) is divided such that half of the traffic travels over back plane #1 600 and half the traffic travels over back plane #2 602 .
- all loop buses 402 that make up the controller loop described in FIG. 4 are similarly divided between back plane #1 600 and back plane #2 602 .
- each mirror bus 400 carrying mirror traffic between two mirrored controllers in each of the controller pairs 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), and 208 ( 4 ) is likewise divided such that half of the traffic travels over back plane #1 600 and half the traffic travels over back plane #2 602 .
- FIG. 7 illustrates the always-on controller interconnect structure of FIG. 6 during operation while one of the two back planes 600 , 602 has been removed.
- the dual back plane configuration interconnect structure of FIG. 6 permits the RAID storage array 102 to remain on-line and operational while either one of the two back planes is faulted or removed for repair.
- failure detection and rerouting circuitry 232 on each controller automatically detects failed links in the controller interconnect structure. Once the hardware circuit 232 detects that a portion of a link or bus ( 400 , 402 ) is no longer operational, it will fail-over to the working portion. As illustrated in FIG.
- back plane #2 602 causes the failure detection and routing circuitry 232 on each controller board 208 to fail over to using the operational half of each bus that is still being carried over back plane #1 600 .
- the remaining back plane i.e., back plane #1 600
- back plane #1 600 continues to provide all of the controller-to-controller communications and data flow that takes place under normal operating conditions. Therefore, the disk array 102 can remain on-line and operational.
- each controller interconnect structure is flexible to accommodate additional or fewer controller pairs 208 .
- the controller interconnect structure can have as few as 2 controller pairs, or as many as 16, 32, or more controller pairs 208 operatively coupled in the same general interconnect configuration as shown in FIGS. 4, 5, 6 and 7 . Increasing the number controller pairs 208 beyond those shown in FIGS. 4, 5, 6 and 7 would involve extending the controller loops in these configurations.
- FIG. 8 illustrates another embodiment of an always-on controller interconnect structure that is suitable for implementation in the RAID storage array 102 of FIGS. 1 and 2.
- the interconnect structure in the FIG. 8 embodiment enables an increase in the number of controller pairs 208 through the introduction of additional controller loops. This is accomplished in general by adding one or more levels of controllers to those already present in the embodiments described above relating to FIGS. 4, 5, 6 and 7 .
- Increasing the number of controllers by adding “levels” allows the average path length between any two controllers to be shorter than if controllers were to be added by extending a single controller loop.
- Adding controller “levels” also adds multiple re-routing paths that can be used to allow multiple interconnect failures while keeping full interconnectivity.
- Each controller level in the multi-level controller interconnect structure of FIG. 8 is configured in a manner similar to that of the controller interconnect structure of FIG. 4 described above. Therefore, like the controller interconnect structure of FIG. 4, a first controller level 800 of FIG. 8 includes controller pairs 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), and 208 ( 4 ) coupled together as mirrored controller pairs and a continuous controller loop formed by coupling of all of the individual controllers from the controller pairs. In addition, however, the controller interconnect structure of FIG. 8 includes one or more additional controller levels such as, for example, level 2 , 802 . Each additional controller level is configured like the first controller level 800 .
- mirror buses 400 carry mirror traffic between the two controllers in each of the mirrored controller pairs (e.g., controller pairs 208 ( 1 ), 208 ( 2 ), 208 ( 3 ), 208 ( 4 ), 208 ( 5 ), 208 ( 6 ), 208 ( 7 ), and 208 ( 8 ) of FIG. 8).
- each controller on a given controller level (e.g., level 1 800 , level 2 802 ) forms part of a continuous controller loop by virtue of being coupled via loop buses 402 (represented by dashed-lined arrows) to two logically adjacent, but non-mirrored controllers within the same level.
- the controller interconnect structure for each controller level is configured like the controller interconnect structure described above with respect to FIG. 4.
- Controller levels in the multi-level interconnect structure of FIG. 8, such as levels 800 and 802 are coupled to one another through loop buses 402 that couple controllers on one controller level to corresponding controllers on another level.
- controller board A 1 from controller pair 208 ( 1 ) on controller level 1 , 800 corresponds with controller board A 3 from controller pair 208 ( 5 ) on controller level 2 , 802 .
- a loop bus 402 operatively couples controller board A 1 208 ( 1 ) to controller board A 3 208 ( 5 ). Therefore, in addition to enabling the controller-to-controller communications described above with respect to the interconnect structure of FIG.
- the multi-level controller interconnect structure of FIG. 8 enables pair-to-pair communications between controller pairs residing on different controller levels.
- the interconnect structure of FIG. 8 provides the same failure detection and rerouting features through failure detection and rerouting circuits 232 .
- FIG. 9 An example method for maintaining controller communications over an always-on controller interconnect structure in a multi-controller RAID storage array 102 will now be described with primary reference to FIG. 9. The method applies generally to the exemplary embodiments discussed above with respect to FIGS. 1 - 8 .
- FIG. 9 is a flow diagram that shows an example of a general method of 5 controller communication performed over an always-on controller interconnect structure in a multi-controller RAID storage array 102 .
- the elements of the described method may be performed by any appropriate means, such as by the execution of processor-readable instructions defined on a processor-readable media, including a disk, a ROM or other such memory device.
- data is received at a first controller in a multi-controller storage array such as RAID storage array 102 .
- a controller pair that is the destination for the data is determined based on a mapping of the data's host address to an array address. Based on the mapping, a packet heading (or “header”) is assigned to the data.
- instruction(s) associated with the data header are accessed in routing register(s) 230 .
- the data is sent over a controller loop in a first direction that is determined by routing register 230 instruction(s) for the associated header information.
- this first direction will take the data to the nearest mirrored controller belonging to the destination controller pair.
- a failure is detected in the controller loop.
- the data is automatically rerouted or “bounced” in a second direction around the controller loop that avoids the detected failure.
- information regarding the loop failure is shared with other controllers so they can reprogram their routing registers to avoid the failure.
- routing registers are reprogrammed with new routing information based on the detected failure.
- the new routing information is used to send additionally received data in a direction over the controller loop that avoids the detected failure.
Abstract
Description
- The present disclosure relates to disk arrays, and more particularly, to a controller interconnect structure within multi-controller disk arrays that permits continued communication between controllers under various failure scenarios.
- Modem mass storage systems continue to provide increasing storage capacities to meet user demands from host computer system applications. A growing reliance on large capacity mass storage has fueled a corresponding demand for enhanced reliability of such storage systems. One popular solution to the demands for increased storage capacity and reliability is the use of multiple smaller storage modules configured in geometries that permit redundancy of stored data to assure data integrity in case of various failures.
- RAID (redundant array of independent disks) disk arrays are an example of a fault tolerant, mass storage technology that has developed in response to the ever-increasing demands for greater storage capacity and reliability. RAID disk arrays supply host computer systems with large amounts of storage capacity in addition to providing redundancy of stored data to assure data integrity in case of various failures. Such disk arrays therefore typically include redundant components such as controllers and power supplies, as well as hot-swap capabilities for various subsystem modules (i.e., an ability to change-out modules without powering down the system).
- Conventional RAID arrays commonly have two controllers that manage the array and perform mirrored memory operations for data redundancy. The controllers make the array appear to the host computer as a single, highly reliable, high capacity disk drive. Both controllers have independent access to all data cache information, all input/output (I/O) state information, and all system state information so that a failure of one of the controllers does not prevent the remaining working controller from accessing all the necessary information to take over sole operation of the array. Significant bandwidth is required on controller interconnect buses to allow the controllers to transfer the necessary information for processing host I/O requests and performing mirrored memory operations.
- As disk arrays become larger, controller pairs can be added to the arrays to increase their computing resources and maintain or improve system performance. However, as the number of controller pairs increases, the amount of data flowing between controllers over the controller interconnect buses increases dramatically. As an example, when a controller pair “A” receives a host computer write command that is destined for the cache memory on controller pair “B”, the controller interconnect not only carries mirrored data traffic and inter-processor communications between controllers in pair “B”, but it also carries pair-to-pair traffic between the two controller pairs “A” and “B”.
- First, the interconnect must carry the traffic from the controller board in pair “A” that received the host data to the controller board in pair “B” that is the destination of the host data. Second, the interconnect must carry the mirror traffic between the two controller boards that form controller pair “B”. Therefore, an increase in the number of controller pairs within a disk array can contribute to performance bottlenecks due to bandwidth limitations of the controller interconnect buses.
- Another consideration regarding controller interconnects is emerging technologies that allow for wider interfaces between disk arrays and host systems. As higher performance host computer connections are developed for connecting RAID storage arrays to host computer systems, controller interconnect buses experience a corresponding increase in the amount of data flowing between controllers within an array. Again, bandwidth limitations on controller interconnect buses within the array can result in performance bottlenecks.
- Another problem that results from adding more controllers to a disk array (i.e. clustering the controllers) is that more and more data travels to remote controllers rather than a local controller in the mirrored controller pair where the data is received. Where there are only 2 controller boards in a disk array, all the host computer disk traffic is destined for the local mirrored cache because there is only one mirrored cache. However, when there are 4 controller boards in an array, the percentage of data flowing to the local mirrored cache drops to 50%. Half the traffic stays with the local cache memory while the other half is destined for the remote pair's cache memory. With 16 pairs of controllers, only about 7% of the traffic is local. Thus, the characteristics of the controller interconnect changes dramatically with the clustering of controllers in the disk array.
- Another important consideration regarding communications between controllers in a clustered disk array is the effect that failures in the controller interconnect have on the operability of the array. Currently, a failure in a controller interconnect can result in a failure in the operation of related array elements. In order to avoid a permanent lock-up of the disk array under such circumstances, various timeout functions must be designed and built into the array hardware. This causes difficulties in hardware design and also increases the complexity of firmware that must be able to tolerate the loss of controller communications without notice.
- Accordingly, the need exists for a controller interconnect structure in disk arrays having clustered controllers that provides for the efficient use of current and future interconnect bandwidth capabilities and that enables continued controller-to-controller communications and disk array operability under various interconnect failure scenarios.
- A controller interconnect structure permits low latency/high bandwidth communications through mirror buses that couple controllers together as mirrored controller pairs within a RAID disk array having a plurality of mirrored (i.e. clustered) controller pairs. The interconnect structure also forms a controller loop that couples controllers together through loop buses. The controller loop provides an automatic fail-over function that enables continued communications between controller pairs in the event that a failure occurs within the controller loop.
- In a first embodiment, a disk array includes at least two pairs of controllers. Each controller pair has a first and second controller that perform mirrored memory operations through a mirror bus that carries mirror data traffic between the two mirrored controllers in the controller pair. In addition, a controller loop is formed by the interconnection of all the controllers through a plurality of loop buses. Each controller is coupled to two logically adjacent controllers through a loop bus such that a continuous loop of controllers is formed. The controller loop permits data and control information to travel in both directions along the loop between the two or more controller pairs. Routing logic in each controller controls the flow of data in the loop such that data packets are normally routed to the nearest mirrored controller associated with array addresses in the data packet headings.
- A failure in the controller loop causes the loop to fail into a single string of controllers. Although the controller loop has failed, the controller string is capable of providing continued communication between all controller pairs. Hardware circuitry on each controller provides an automatic fail-over function that tolerates failures in the controller loop by detecting a failure and rerouting data in a different direction to avoid the failure. Therefore, data initially traveling in one direction through the loop will be rerouted or “bounced” in the opposite direction when a loop failure is encountered. The controller string then carries the data to its destination controller pair.
- Another embodiment includes the controllers configured as in the prior embodiment, coupled together through two back plane interconnect boards. One half of each mirror bus and one half of each loop bus runs through each of the two back planes. Both halves of each bus can work in unison under normal operation or one half of each bus is able to take over all the data traffic of both halves in the event of a failure condition. Thus, the dual back plane configuration permits on-line repair of either back plane. Hardware circuitry on controller boards automatically detects failed links between boards. A detected failure on any bus automatically fails the bus over to using the operational half of the bus. Thus, either one of the two back planes can be removed and repaired while data continues flowing between controllers over the operational half of each bus that runs through the remaining back plane.
- A third embodiment includes two or more sets of controllers generally configured as in the prior embodiments and logically coupled into levels. The embodiment allows the size of a disk array system to be scaled up significantly by expanding the number of controllers through additional controller loops. Each loop of controllers is configured as a level of controllers stacked upon another level of controllers. Each additional controller loop has the same properties as in the previously described embodiments where broken or failed links do not disable the transfer of data through the system.
- In another embodiment, controllers include programmable routing registers that contain routing information to control the direction of data flow along a controller loop. The routing registers permit a matched data flow along loop bus segments so that no single loop bus segment is over burdened. When a controller receives a host computer I/O command, the controller's routing logic is configured to access the routing information from the programmable routing register to determine the direction in which to send the I/O command data.
- In yet another embodiment, routing logic is configured to reprogram routing registers when a failure occurs in a controller loop so that data flow between controllers is more efficient. Hardware circuitry automatically detects a failure and reroutes data to avoid the failure. However, continually sending data in one direction and then rerouting it by a “hardware” reroute is not the most efficient use of the interconnect structure's capacity. Therefore, when hardware detects a failure and reroutes data, it also notifies the routing logic of the failure so routing registers will be reprogrammed to provide modified data routes that avoid the failure without traversing the less efficient hardware reroute. Data is thus initially routed in a direction that avoids the failure. The hardware detection circuitry can also be configured to reprogram routing registers.
- The same reference numbers are used throughout the drawings to reference like components and features.
- FIG. 1 illustrates a system environment that is suitable for implementing an arrayed storage device having an always-on controller interconnect structure.
- FIG. 2 is a block diagram illustrating in greater detail, a particular embodiment of the system environment of FIG. 1 including a host computer device and an arrayed storage device implemented as a RAID disk array having an always-on controller interconnect structure.
- FIG. 3 is a block diagram illustrating in greater detail, a controller pair such as the controller pair illustrated in the block diagram of FIG. 2.
- FIG. 4 shows a logical representation of a controller interconnect structure such as might be implemented in the RAID disk array of FIG. 2.
- FIG. 5 illustrates the controller interconnect structure of FIG. 4 under a particular failure scenario.
- FIG. 6 illustrates a controller interconnect structure such as that of FIG. 4 in a redundant back plane configuration.
- FIG. 7 illustrates the controller interconnect structure of FIG. 6 under a particular repair scenario.
- FIG. 8 shows a logical representation of another embodiment of a controller interconnect structure having an additional controller interconnect level such as might be implemented in the RAID disk array of FIG. 2.
- FIG. 9 is a flow diagram illustrating an example of a general method of performing controller communications over an always-on controller interconnect structure.
- A controller interconnect structure within a RAID disk array enables continuous low latency/high bandwidth communications between a plurality of controller pairs within the array. Mirror buses carry high speed mirror traffic between mirrored controllers performing mirrored memory operations. Loop buses carry inter-processor communications and other traffic between controller pairs coupled together in a controller loop. Benefits of the interconnect structure include an ability to support continued controller communications and online disk array operations under various failure and repair conditions that might otherwise render a disk array inoperable. In addition, the controller interconnect structure provides for easy expansion of the number of controllers within disk arrays as arrays continue to be scaled up in size to meet increasing storage demands from user host systems.
- Exemplary System Environment For Implementing An Always-On Controller Interconnect Structure
- FIG. 1 illustrates a
system environment 100 suitable for implementing an always-on controller interconnect structure. Thesystem 100 includes arrayed storage device (e.g. a RAID storage array) 102 operatively coupled to host device(s) 104 throughnetwork 106.Storage device 102 typically provides for multipleredundant network connections 106.Network connection 106 can include, for example, a LAN (local area network), a WAN (wide area network), an intranet, the Internet, a fiber optic cable link, a wireless link, a direct connection, or any other suitable communication link. Host device(s) 104 can be implemented as a variety of general purpose computing devices including, for example, a personal computer (PC), a laptop computer, a server, a Web server, and other devices configured to communicate with arrayedstorage device 102. - Although embodiments of arrayed
storage device 102 are disclosed herein as RAID storage arrays, the arrayedstorage device 102 is not limited in this regard. Accordingly, this disclosure is applicable to other configurations of arrayed storage components as currently exist or as might exist in the future that include different array architectures intended to offer high-performance, fault-tolerant mass storage similar to that provided by currently available RAID systems. Therefore, arrayedstorage device 102 more generally refers to a plurality of storage components/devices operatively coupled in an array for the general purpose of increasing storage performance. Storage performance goals typically include storage capacity, low cost per stored megabyte, high input/output performance, and high data availability through redundancy and fault tolerance. Storage components/devices operatively coupled within arrayedstorage devices 102 may include devices such as magnetic disk drives, tape drives, optical read/write disk drives, solid state disks and the like. Such storage components are generally well known in the art of data storage technology. - Exemplary Embodiment Of A System For Implementing An Always-On Controller Interconnect Structure
- FIGS. 2 and 3 are block diagrams illustrating a particular embodiment of a
host computer device 104 and an arrayedstorage device 102 as might be implemented in thesystem environment 100 of FIG. 1. The arrayedstorage device 102 of FIG. 1 is embodied in FIG. 2 as aRAID storage array 102 having a plurality of clustered controller pairs 208.Host device 104 is embodied generally as a computer such as a personal computer (PC), a laptop computer, a server, a Web server, or other computer device configured to communicate withRAID storage array 102. -
Host device 104 typically includes aprocessor 200, a volatile memory 202 (i.e., RAM), and a nonvolatile memory 204 (e.g., ROM, hard disk, floppy disk, CD-ROM, etc.).Nonvolatile memory 204 generally provides storage of computer readable instructions, data structures, program modules and other data forhost device 104.Host device 104 may implementvarious application programs 206 stored inmemory 204 and executed onprocessor 200 that create or otherwise access data to be transferred vianetwork connection 106 toRAID storage array 102 for storage and subsequent retrieval.Such applications 206 might include software programs implementing, for example, word processors, databases, spread sheets, browsers, multimedia players, illustrators, computer-aided design tools and the like. Thus,host device 104 provides a regular flow of data I/O requests to be serviced byRAID storage array 102. -
RAID storage array 102 is generally designed to provide continuous data storage and data retrieval for computer devices such as host device(s) 104, and to do so under various fault conditions that may occur. Thus,RAID array 102 typically includes redundant subsystems such as controller pairs 208 and power andcooling subsystems 210 that permit continued access to theRAID array 102 even during a failure of one of the subsystems. In addition,RAID array 102 typically provides hot-swapping capabilities for array components (i.e. the ability to remove and replace components while thearray 102 remains online) such as the controllers in controller pairs 208, the power/coolingsubsystems 210, and the disk drives 214 in the array ofdisks 212. - Each controller pair on
RAID array 102 includes a first controller (e.g., CTLR A1) and a second controller (e.g., CTLR A2). The two controllers in eachcontroller pair 208 mirror each other and are generally configured to redundantly store and access data on disk drives 214. Thus, controllers A1 and A2 perform tasks such as mapping host data to disk drives, performing RAID calculations, mirroring data between redundant controller boards, attaching validation tags to data before saving the data todisk drives 214 and checking the tags to ensure data from adisk drive 214 is correct before sending the data back to ahost device 104. Controllers in eachcontroller pair 208 also tolerate faults such asdisk drive 214 failures by recreating data that may be lost during such failures. - FIG. 3 is a block diagram illustrating an example of a controller Al from a controller pair208(1) in more detail. Controller A2 from controller pair 208(1) is represented in FIG. 3 but is not specifically detailed because it is configured the same as controller Al. Referring to controller Al as a representative controller example, each controller in a
controller pair 208 onRAID array 102 typically includes I/O processor(s) such as FC (fiber channel) I/O processor(s) 216, main processor(s) 218, nonvolatile (NV)RAM 220, memory 222 (e.g., ROM, RAM), and one or more ASICs (application specific integrated circuits) such asmemory control ASIC 224.NV RAM 220 is typically supported by a battery backup (not shown) that preserves data inNV RAM 220 in the event power is lost to controller(s) 208.Memory 222 generally provides storage of computer readable instructions, data structures, program modules and other data forRAID storage array 102. Accordingly,nonvolatile memory 222 includesfirmware 226 which is generally configured to execute on processor(s) 218 and supportnormal disk array 102 operations.Firmware 226 is also typically configured to handle various fault scenarios that may arise inRAID array 102. - As is more fully discussed herein below, routing
logic 228 and routing register(s) 230 are configured to route data between various controller pairs 208 via a controller interconnect structure. Also discussed more fully below is a hardware detection and reroutingcircuit 232 that is generally configured to detect controller interconnect failures and reroute data in order to circumvent such failures. - FC I/O processor(s)216 on controllers (e.g., controller Al of FIG. 3) receives data and commands from
host device 104 vianetwork connection 106. FC I/O processor(s) 216 communicates with main processor(s) 218 through standard protocols and interrupt procedures to transfer data and commands to redundant controllers (e.g., controller A2 of FIG. 3) and generally move data betweenNV RAM 220 andvarious disk drives 214 to ensure that data is stored redundantly. -
Memory control ASIC 224 generally controls data storage and retrieval, data manipulation, redundancy management, and the like through communications between mirrored controllers such as controllers Al and A2 of FIG. 3, for example.Memory controller ASIC 224 handles mirroring of data between controllers, tagging of data sectors being striped todisks 214 in the array ofdisks 212 and RAID calculations to write parity information across the disk drives 214, as well as data reconstruction in the event of a disk drive failure. Data striping and parity checking are well-known to those skilled in the art.Memory control ASIC 224 also typically includes internal buffers (not shown) that facilitate testing ofmemory 222 to ensure that all regions of mirrored memory (e.g., between mirrored controllers A1 and A2) are compared to be identical and checked for ECC (error checking and correction) errors on a regular basis.Memory control ASIC 224 notifiesprocessor 218 of these and other errors it detects.Firmware 226 is configured to manage errors detected bymemory control ASIC 224 in a tolerant manner which may include, for example, preventing the corruption ofarray 102 data or working around a detected error/fault through a redundant subsystem to prevent theRAID array 102 from crashing. - Exemplary Embodiments Of An Always-On Controller Interconnect Structure
- FIG. 4 illustrates an example of an always-on controller interconnect structure that is suitable for implementation in the
RAID storage array 102 of FIGS. 1 and 2. FIG. 4 includes a plurality of controller pairs 208 interconnected through a network of mirror buses 400 (represented by solid-lined arrows) and loop buses 402 (represented by dashed-lined arrows). Each controller pair 208(1), 208(2), 208(3), and 208(4) includes a first and second controller operatively coupled through a first interconnect, ormirror bus 400. For example, controller A1 is coupled to controller A2 through amirror bus 400, controller B1 is coupled to controller B2 through anothermirror bus 400, and so on. Themirror buses 400 carry mirror traffic between the two controllers in each of the mirrored controller pairs 208(1), 208(2), 208(3), and 208(4). Themirror buses 400 between eachcontroller pair 208 are therefore capable of providing low latency/high bandwidth transfers as host data and RAID maps are stored and accessed in mirrored memory (i.e. NV RAM 220). - Each controller in the controller interconnect structure of FIG. 4 is additionally coupled to two other logically adjacent controllers through a second interconnect, called
loop buses 402. The two logically adjacent controllers coupled to a particular controller throughloop buses 402 do not include that particular controller's mirrored controller, which is already coupled through amirror bus 400. Thus, controller B1, for example, is coupled to logically adjacent controllers A1 and C1 throughloop buses 402. - The controller interconnect structure includes two points where
loop buses 402 cross over between first controllers from the controller pairs 208 to second controllers from the controller pairs 208. The cross over forms a connection between a row of first controllers (i.e., A1, B1, C1, and D1) and a row of second controllers (i.e., A2, B2, C2 and D2), which in turn forms a continuous loop of controllers. Referring to the controller interconnect structure of FIG. 4, aloop bus 402 crosses over to couple first controller A1 to logically adjacent second controller D2. Similarly, aloop bus 402 crosses over to couple first controller D1 to logically adjacent second controller A2. Thus, each controller forms part of a continuous controller loop by virtue of being coupled to two logically adjacent, but non-mirrored, controllers. - Whereas
mirror buses 400 typically carry mirror traffic between two controllers within a mirrored controller pair (e.g., 208(1)), theloop buses 402 carry traffic between the various controller pairs. Pair-to-pair traffic, or “loop traffic”, includes data received at one controller (e.g., controller A1 of controller pair 208(1)) that is destined for another pair of mirrored controllers (e.g., controller pair 208(3)) in addition to all IPC (inter-processor communication) traffic. Pair-to-pair traffic flows in both directions around the controller loop. - As mentioned briefly above, routing
logic 228 and routing register(s) 230 (see FIG. 3) are configured to route data between various controller pairs 208 via the controller interconnect structure. In general, therouting logic 228 routes data along the continuous controller loop (see FIG. 4) so that data arrives at its destination via the quickest route and, so that each segment of the controller loop is used efficiently. Therouting logic 228 determines the best direction to send data along the controller loop based on information/instructions from routing register(s) 230 and on the array's mapping of host addresses to array addresses. A controller that receives the data assigns a header or data packet heading that identifies whichcontroller pair 208 is the proper destination for the data. Therouting logic 228uses routing register 230 instructions associated with the data header to send the data in a direction along the controller loop which typically, but not necessarily, takes the data to the nearest mirrored controller of the destination controller pair. For example, data received from ahost 104 by controller A1 208(1) that is destined for controller pair B 208(2) will be routed to the right, over thesingle loop bus 402 segment between controllers A1 and B1. Thus, the data is routed to the nearest mirrored controller B1 of the destination controller pair B 208(2). - Routing register(s)230 are programmable registers located in the
routing logic 228 that provide therouting logic 228 with information on which direction to send data destined for acontroller pair 208. Routing register(s) 230 are initially programmed, for example, byprocessor 218 to contain information that therouting logic 228 uses to determine which direction to route data over the controller loop (see FIG. 4). - Under certain circumstances, the nearest mirrored controller of a
destination controller pair 208 may be equidistant from the controller sending the data. For example, referring to FIG. 4, data received from ahost device 104 by controller A1 208(1) that is destined for controller pair C 208(3), is an equal distance away from the nearest mirrored controller of destination controller pair C 208(3). That is, it is no closer for therouting logic 228 on controller A1 to send the data to controller C1 208(3) by way of controller B1 208(2) than it is to send the data to controller C2 208(3) by way of controller D2 208(4). Assuming workloads from host device(s) 104 are spread evenly among the various controllers of FIG. 4, the programmable routing registers 230 permit an evenly matched flow of data over allloop bus 402 segments of the controller loop. Thus, in the case where the nearest mirrored controller of adestination controller pair 208 is equidistant from the controller sending the data, routing registers 230 may tellrouting logic 228 on each controller to send such equidistant data in the same direction around the controller loop, such as to the right, or clockwise. Thus, when data at controller A1 208(1) is destined for controller pair C 208(3), it is sent to controller C1 208(3) by way of controller B1 208(2). Likewise, when data at controller B1 208(2) is destined for controller pair D 208(4), it is sent to controller D1 208(4) by way of controller C1 208(3), instead of being sent to controller D2 208(4) by way of controller A1 208(1). The result is that theloop bus 402 segment between controller A1 208(1) and controller B1 208(2) does not get overburdened by excess traffic. In addition, in the aggregate, when workloads from host device(s) 104 are spread evenly among the various controllers, the flow of data over each segment of the controller loop is evenly matched. - As mentioned briefly above with reference to FIGS. 2 and 3, a failure detection and rerouting
circuit 232 is configured to detect failures in the controller interconnect structure and to reroute data in order to circumvent the failures. FIG. 5 illustrates the controller interconnect structure of FIG. 4 discussed above and includes an example of afailure 500 in the controller loop at a particular point in theloop bus 402. It is apparent from FIG. 5 that a break in the controller loop interconnect structure causes the continuous controller loop to fail into a controller string. Thus, anyfailure 500 in the controller loop will mark the end points of the controller string. The endpoints of the controller string of FIG. 5 are at controller C1 208(3) and controller B1 208(2). The detection/rerouting hardware circuits 232 on controllers C1 208(3) and B1 208(2) automatically detect theloop failure 500 and “bounce” data that encounters thefailure 500 back in the opposite direction. For example, if data is traveling from controller A1 208(1) through controller B1 208(2) to controller C1 208(3), the detection/reroutinghardware circuit 232 on controller B1 208(2) will “bounce” the data back over the controller string so it arrives at controller C1 208(3) by way of controller D1 208(4). - In addition to detecting and rerouting or “bouncing” data around a
failure 500, thehardware circuits 232 can provide notification to processor(s) 218 of the failure so that the processor(s) 218 can reprogram the routing registers 230 on the controllers. This enables therouting logic 228 to avoid thefailure 500 when it initially routes data over the controller interconnect structure. Reprogramming the routing registers 230 in this manner makes more efficient use of the controller interconnect under a failure condition. Alternatively, thehardware circuits 232 may themselves modify the routing information in the routing registers 230 under such failure conditions. - Under certain circumstances,
mirror buses 400, which typically carry mirror traffic between two controllers within a mirrored controller pair (e.g., 208(1)), can also be used to carry “loop traffic”. For example, in a “partially populated” back-plane configuration where a controller pair is not present, “loop traffic” data may be routed over amirror bus 400 between a mirrored controller pair in order to avoid the non-present controllers while still maintaining a full controller loop interconnection. Therefore, the controller loop may be formed using bothloop buses 402 andmirror buses 400. Under these circumstances,hardware circuits 232 would provide some low-level physical presence information to therouting logic 228 that will change the way traffic is routed through the controller loop. - FIG. 6 illustrates another embodiment of the always-on controller interconnect structure suitable for implementation in the
RAID storage array 102 of FIGS. 1 and 2. The interconnect structure of FIG. 6 is configured like the FIG. 4 interconnect structure described above. Thus,mirror buses 400 carry mirror traffic between the two controllers in each of the mirrored controller pairs 208(1), 208(2), 208(3), and 208(4), and each controller forms part of a continuous controller loop by virtue of being coupled, byinterconnect 402, to two logically adjacent, but non-mirrored controllers. In addition, however, the FIG. 6 embodiment includes a dual back plane configuration that allows for the on-line repair of a failed back plane while the remaining back plane continues to provide a fully functioning interconnect structure between all controllers in theRAID storage array 102. - In the FIG. 6 embodiment, each controller from controller pairs208(1), 208(2), 208(3), and 208(4) is coupled to or plugged into two separate interconnects. The interconnects are embodied as
back planes Back plane # 1 600 and backplane # 2 602 both carry one half of each bus that is shown in the controller interconnect structure of FIG. 4. In addition, each half of each bus (i.e.,mirror buses 400 and loop buses 402) is bi-directional. Therefore, theloop bus 402 that carries pair-to-pair traffic from controller A1 208(1) to controller D1 208(4) is divided such that half of the traffic travels overback plane # 1 600 and half the traffic travels overback plane # 2 602. Furthermore, allloop buses 402 that make up the controller loop described in FIG. 4 are similarly divided betweenback plane # 1 600 and backplane # 2 602. Moreover, eachmirror bus 400 carrying mirror traffic between two mirrored controllers in each of the controller pairs 208(1), 208(2), 208(3), and 208(4) is likewise divided such that half of the traffic travels overback plane # 1 600 and half the traffic travels overback plane # 2 602. - It will be apparent to those skilled in the art that the back planes described in the embodiment of FIG. 6 are not the only physical interconnect medium over which buses can be divided. Thus, the embodiment of FIG. 6 is illustrated by way of example rather than by way of limitation. For example, cables might be used as the physical interconnect medium over which the buses are divided in halves. Thus, the removal or failure of one cable carrying half of a bus would not prevent data from flowing over a cable carrying the other operational half of the bus.
- FIG. 7 illustrates the always-on controller interconnect structure of FIG. 6 during operation while one of the two
back planes RAID storage array 102 to remain on-line and operational while either one of the two back planes is faulted or removed for repair. As discussed above with respect to FIG. 5, failure detection and reroutingcircuitry 232 on each controller automatically detects failed links in the controller interconnect structure. Once thehardware circuit 232 detects that a portion of a link or bus (400, 402) is no longer operational, it will fail-over to the working portion. As illustrated in FIG. 7, removal ofback plane # 2 602 causes the failure detection androuting circuitry 232 on eachcontroller board 208 to fail over to using the operational half of each bus that is still being carried overback plane # 1 600. The remaining back plane (i.e., backplane # 1 600) continues to provide all of the controller-to-controller communications and data flow that takes place under normal operating conditions. Therefore, thedisk array 102 can remain on-line and operational. - Although the always-on controller interconnect structures described above with reference to FIGS. 4, 5,6 and 7 include 4 controller pairs, the interconnect structures are not limited in this regard. Specifically, each controller interconnect structure is flexible to accommodate additional or fewer controller pairs 208. For example, the controller interconnect structure can have as few as 2 controller pairs, or as many as 16, 32, or more controller pairs 208 operatively coupled in the same general interconnect configuration as shown in FIGS. 4, 5, 6 and 7. Increasing the number controller pairs 208 beyond those shown in FIGS. 4, 5, 6 and 7 would involve extending the controller loops in these configurations.
- FIG. 8 illustrates another embodiment of an always-on controller interconnect structure that is suitable for implementation in the
RAID storage array 102 of FIGS. 1 and 2. Although as just mentioned above, the number of controllers can be increased in the previously described controller interconnect structures by extending the controller loops, the interconnect structure in the FIG. 8 embodiment enables an increase in the number of controller pairs 208 through the introduction of additional controller loops. This is accomplished in general by adding one or more levels of controllers to those already present in the embodiments described above relating to FIGS. 4, 5, 6 and 7. Increasing the number of controllers by adding “levels” allows the average path length between any two controllers to be shorter than if controllers were to be added by extending a single controller loop. Adding controller “levels” also adds multiple re-routing paths that can be used to allow multiple interconnect failures while keeping full interconnectivity. - Each controller level in the multi-level controller interconnect structure of FIG. 8 is configured in a manner similar to that of the controller interconnect structure of FIG. 4 described above. Therefore, like the controller interconnect structure of FIG. 4, a
first controller level 800 of FIG. 8 includes controller pairs 208(1), 208(2), 208(3), and 208(4) coupled together as mirrored controller pairs and a continuous controller loop formed by coupling of all of the individual controllers from the controller pairs. In addition, however, the controller interconnect structure of FIG. 8 includes one or more additional controller levels such as, for example,level first controller level 800. - Accordingly, for each controller level (e.g.,
level 1 800,level 2 802, etc.), mirror buses 400 (represented by solid-lined arrows) carry mirror traffic between the two controllers in each of the mirrored controller pairs (e.g., controller pairs 208(1), 208(2), 208(3), 208(4), 208(5), 208(6), 208(7), and 208(8) of FIG. 8). In addition, each controller on a given controller level (e.g.,level 1 800,level 2 802) forms part of a continuous controller loop by virtue of being coupled via loop buses 402 (represented by dashed-lined arrows) to two logically adjacent, but non-mirrored controllers within the same level. Thus, the controller interconnect structure for each controller level is configured like the controller interconnect structure described above with respect to FIG. 4. - Controller levels in the multi-level interconnect structure of FIG. 8, such as
levels loop buses 402 that couple controllers on one controller level to corresponding controllers on another level. For example, controller board A1 from controller pair 208(1) oncontroller level controller level loop bus 402 operatively couples controller board A1 208(1) to controller board A3 208(5). Therefore, in addition to enabling the controller-to-controller communications described above with respect to the interconnect structure of FIG. 4 (i.e., mirrored communications between controllers in a mirrored pair, and pair-to-pair communications between different controller pairs), the multi-level controller interconnect structure of FIG. 8 enables pair-to-pair communications between controller pairs residing on different controller levels. In addition, like the controller interconnect structure of FIG. 4, the interconnect structure of FIG. 8 provides the same failure detection and rerouting features through failure detection and reroutingcircuits 232. - Exemplary Method For Maintaining Controller Communications Over An Always-On Controller Interconnect Structure
- An example method for maintaining controller communications over an always-on controller interconnect structure in a multi-controller
RAID storage array 102 will now be described with primary reference to FIG. 9. The method applies generally to the exemplary embodiments discussed above with respect to FIGS. 1-8. - FIG. 9 is a flow diagram that shows an example of a general method of5 controller communication performed over an always-on controller interconnect structure in a multi-controller
RAID storage array 102. The elements of the described method may be performed by any appropriate means, such as by the execution of processor-readable instructions defined on a processor-readable media, including a disk, a ROM or other such memory device. - Referring to the method illustrated in FIG. 9, at
block 900, data is received at a first controller in a multi-controller storage array such asRAID storage array 102. Atblock 902, a controller pair that is the destination for the data is determined based on a mapping of the data's host address to an array address. Based on the mapping, a packet heading (or “header”) is assigned to the data. Atblock 903, instruction(s) associated with the data header are accessed in routing register(s) 230. Atblock 904, the data is sent over a controller loop in a first direction that is determined by routingregister 230 instruction(s) for the associated header information. Typically, this first direction will take the data to the nearest mirrored controller belonging to the destination controller pair. Atblock 906, a failure is detected in the controller loop. Atblock 908, the data is automatically rerouted or “bounced” in a second direction around the controller loop that avoids the detected failure. Atblock 909, information regarding the loop failure is shared with other controllers so they can reprogram their routing registers to avoid the failure. Atblock 910, routing registers are reprogrammed with new routing information based on the detected failure. Atblock 912, the new routing information is used to send additionally received data in a direction over the controller loop that avoids the detected failure. - Although the description above uses language that is specific to structural features and/or methodological acts, it is to be understood that the invention defined in the appended claims is not limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the invention.
- Additionally, while one or more methods have been disclosed by means of flow diagrams and text associated with the blocks of the flow diagrams, it is to be understood that the blocks do not necessarily have to be performed in the order in which they were presented, and that an alternative order may result in similar advantages.
Claims (40)
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/146,546 US20030217211A1 (en) | 2002-05-14 | 2002-05-14 | Controller communications over an always-on controller interconnect |
JP2003104750A JP2003330626A (en) | 2002-05-14 | 2003-04-09 | Controller communication over always-on controller interconnect |
DE10317925A DE10317925B4 (en) | 2002-05-14 | 2003-04-17 | Control communication via a constantly switched on control connection |
GB0309915A GB2388700B (en) | 2002-05-14 | 2003-04-30 | Controller communications over an always-on controller interconnect |
US11/150,796 US7480815B2 (en) | 2002-05-14 | 2005-06-10 | Controller communications over an always-on controller interconnect |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/146,546 US20030217211A1 (en) | 2002-05-14 | 2002-05-14 | Controller communications over an always-on controller interconnect |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/150,796 Continuation US7480815B2 (en) | 2002-05-14 | 2005-06-10 | Controller communications over an always-on controller interconnect |
Publications (1)
Publication Number | Publication Date |
---|---|
US20030217211A1 true US20030217211A1 (en) | 2003-11-20 |
Family
ID=29269756
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/146,546 Abandoned US20030217211A1 (en) | 2002-05-14 | 2002-05-14 | Controller communications over an always-on controller interconnect |
US11/150,796 Expired - Lifetime US7480815B2 (en) | 2002-05-14 | 2005-06-10 | Controller communications over an always-on controller interconnect |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/150,796 Expired - Lifetime US7480815B2 (en) | 2002-05-14 | 2005-06-10 | Controller communications over an always-on controller interconnect |
Country Status (4)
Country | Link |
---|---|
US (2) | US20030217211A1 (en) |
JP (1) | JP2003330626A (en) |
DE (1) | DE10317925B4 (en) |
GB (1) | GB2388700B (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050172097A1 (en) * | 2004-01-30 | 2005-08-04 | Hewlett-Packard Development Company, L.P. | Storage system with capability to allocate virtual storage segments among a plurality of controllers |
US20060106982A1 (en) * | 2001-09-28 | 2006-05-18 | Dot Hill Systems Corporation | Certified memory-to-memory data transfer between active-active raid controllers |
US20060161709A1 (en) * | 2005-01-20 | 2006-07-20 | Dot Hill Systems Corporation | Safe message transfers on PCI-Express link from RAID controller to receiver-programmable window of partner RAID controller CPU memory |
US20060282701A1 (en) * | 2001-09-28 | 2006-12-14 | Dot Hill Systems Corporation | Method for adopting an orphan i/o port in a redundant storage controller |
US20070288792A1 (en) * | 2003-02-19 | 2007-12-13 | Istor Networks, Inc. | Storage controller redundancy using packet-based protocol to transmit buffer data over reflective memory channel |
US20080005470A1 (en) * | 2006-06-30 | 2008-01-03 | Dot Hill Systems Corporation | System and method for sharing sata drives in active-active raid controller system |
US20080201616A1 (en) * | 2007-02-20 | 2008-08-21 | Dot Hill Systems Corporation | Redundant storage controller system with enhanced failure analysis capability |
US20160191454A1 (en) * | 2011-11-22 | 2016-06-30 | Microsoft Technology Licensing, Llc | Providing network capability over a converged interconnect fabric |
US11327858B2 (en) | 2020-08-11 | 2022-05-10 | Seagate Technology Llc | Preserving data integrity during controller failure |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6990547B2 (en) * | 2001-01-29 | 2006-01-24 | Adaptec, Inc. | Replacing file system processors by hot swapping |
US7266665B2 (en) * | 2003-06-17 | 2007-09-04 | International Business Machines Corporation | Method, system, and article of manufacture for remote copying of data |
JP2007206766A (en) * | 2006-01-31 | 2007-08-16 | Fujitsu Ltd | Data storage system, data storage control device, and failure part diagnostic method |
US7865766B2 (en) * | 2006-07-26 | 2011-01-04 | International Business Machines Corporation | Providing increased availability of I/O drawers during concurrent I/O hub repair |
JPWO2008050456A1 (en) * | 2006-10-27 | 2010-02-25 | 富士通株式会社 | Computer system, data relay device, and computer system control method |
US7669007B2 (en) * | 2007-01-04 | 2010-02-23 | International Business Machines Corporation | Mirrored redundant array of independent disks (RAID) random access performance enhancement |
US7783917B2 (en) * | 2007-02-26 | 2010-08-24 | International Business Machines Corporation | Selection of data arrays |
FR2988931B1 (en) * | 2012-03-30 | 2015-10-16 | Schneider Toshiba Inverter | CONTROL DEVICE EMPLOYED IN A POWER SUPPLY SYSTEM WITH CUTTING |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4354267A (en) * | 1979-09-10 | 1982-10-12 | Hitachi, Ltd. | Data transmission system utilizing loop transmission lines between terminal units |
US5544330A (en) * | 1994-07-13 | 1996-08-06 | Emc Corporation | Fault tolerant interconnect topology using multiple rings |
US5761705A (en) * | 1996-04-04 | 1998-06-02 | Symbios, Inc. | Methods and structure for maintaining cache consistency in a RAID controller having redundant caches |
US6128750A (en) * | 1996-11-14 | 2000-10-03 | Emc Corporation | Fail-over switching system |
US6192027B1 (en) * | 1998-09-04 | 2001-02-20 | International Business Machines Corporation | Apparatus, system, and method for dual-active fibre channel loop resiliency during controller failure |
US6256748B1 (en) * | 1997-04-29 | 2001-07-03 | Bull, S.A. | Method and device for connecting a data processing system central unit to a redundancy data storage subsystem |
US6504817B2 (en) * | 1997-03-31 | 2003-01-07 | Hewlett-Packard Company | Fiber channel arbitrated loop dynamic loop sizing |
US20030065841A1 (en) * | 2001-09-28 | 2003-04-03 | Pecone Victor Key | Bus zoning in a channel independent storage controller architecture |
US6578158B1 (en) * | 1999-10-28 | 2003-06-10 | International Business Machines Corporation | Method and apparatus for providing a raid controller having transparent failover and failback |
US6643795B1 (en) * | 2000-03-30 | 2003-11-04 | Hewlett-Packard Development Company, L.P. | Controller-based bi-directional remote copy system with storage site failover capability |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TW360843B (en) * | 1996-12-02 | 1999-06-11 | Ibm | Shared loop audio/video server system |
JP2001523860A (en) * | 1997-11-14 | 2001-11-27 | 3ウェア、 インコーポレイテッド | High-performance disk array controller |
US6230240B1 (en) * | 1998-06-23 | 2001-05-08 | Hewlett-Packard Company | Storage management system and auto-RAID transaction manager for coherent memory map across hot plug interface |
US6330687B1 (en) * | 1998-11-13 | 2001-12-11 | Digi-Data Corporation | System and method to maintain performance among N single raid systems during non-fault conditions while sharing multiple storage devices during conditions of a faulty host computer or faulty storage array controller |
US6493795B1 (en) * | 1998-12-30 | 2002-12-10 | Emc Corporation | Data storage system |
DE60028793T2 (en) * | 1999-08-24 | 2007-05-24 | Network Appliance, Inc., Sunnyvale | SCALABLE FILE SERVER WITH HIGHLY AVAILABLE PAIRS |
KR100364895B1 (en) * | 2000-06-12 | 2002-12-16 | 아라리온 (주) | Method of controlling data access and system thereof |
-
2002
- 2002-05-14 US US10/146,546 patent/US20030217211A1/en not_active Abandoned
-
2003
- 2003-04-09 JP JP2003104750A patent/JP2003330626A/en not_active Withdrawn
- 2003-04-17 DE DE10317925A patent/DE10317925B4/en not_active Expired - Fee Related
- 2003-04-30 GB GB0309915A patent/GB2388700B/en not_active Expired - Fee Related
-
2005
- 2005-06-10 US US11/150,796 patent/US7480815B2/en not_active Expired - Lifetime
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4354267A (en) * | 1979-09-10 | 1982-10-12 | Hitachi, Ltd. | Data transmission system utilizing loop transmission lines between terminal units |
US5544330A (en) * | 1994-07-13 | 1996-08-06 | Emc Corporation | Fault tolerant interconnect topology using multiple rings |
US5761705A (en) * | 1996-04-04 | 1998-06-02 | Symbios, Inc. | Methods and structure for maintaining cache consistency in a RAID controller having redundant caches |
US6128750A (en) * | 1996-11-14 | 2000-10-03 | Emc Corporation | Fail-over switching system |
US6504817B2 (en) * | 1997-03-31 | 2003-01-07 | Hewlett-Packard Company | Fiber channel arbitrated loop dynamic loop sizing |
US6256748B1 (en) * | 1997-04-29 | 2001-07-03 | Bull, S.A. | Method and device for connecting a data processing system central unit to a redundancy data storage subsystem |
US6192027B1 (en) * | 1998-09-04 | 2001-02-20 | International Business Machines Corporation | Apparatus, system, and method for dual-active fibre channel loop resiliency during controller failure |
US6578158B1 (en) * | 1999-10-28 | 2003-06-10 | International Business Machines Corporation | Method and apparatus for providing a raid controller having transparent failover and failback |
US6643795B1 (en) * | 2000-03-30 | 2003-11-04 | Hewlett-Packard Development Company, L.P. | Controller-based bi-directional remote copy system with storage site failover capability |
US20030065841A1 (en) * | 2001-09-28 | 2003-04-03 | Pecone Victor Key | Bus zoning in a channel independent storage controller architecture |
Cited By (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7536495B2 (en) | 2001-09-28 | 2009-05-19 | Dot Hill Systems Corporation | Certified memory-to-memory data transfer between active-active raid controllers |
US20060106982A1 (en) * | 2001-09-28 | 2006-05-18 | Dot Hill Systems Corporation | Certified memory-to-memory data transfer between active-active raid controllers |
US20060282701A1 (en) * | 2001-09-28 | 2006-12-14 | Dot Hill Systems Corporation | Method for adopting an orphan i/o port in a redundant storage controller |
US7558897B2 (en) | 2001-09-28 | 2009-07-07 | Dot Hill Systems Corporation | Method for adopting an orphan I/O port in a redundant storage controller |
US20070288792A1 (en) * | 2003-02-19 | 2007-12-13 | Istor Networks, Inc. | Storage controller redundancy using packet-based protocol to transmit buffer data over reflective memory channel |
US7305520B2 (en) * | 2004-01-30 | 2007-12-04 | Hewlett-Packard Development Company, L.P. | Storage system with capability to allocate virtual storage segments among a plurality of controllers |
US20050172097A1 (en) * | 2004-01-30 | 2005-08-04 | Hewlett-Packard Development Company, L.P. | Storage system with capability to allocate virtual storage segments among a plurality of controllers |
US20060161709A1 (en) * | 2005-01-20 | 2006-07-20 | Dot Hill Systems Corporation | Safe message transfers on PCI-Express link from RAID controller to receiver-programmable window of partner RAID controller CPU memory |
US7543096B2 (en) * | 2005-01-20 | 2009-06-02 | Dot Hill Systems Corporation | Safe message transfers on PCI-Express link from RAID controller to receiver-programmable window of partner RAID controller CPU memory |
US7536508B2 (en) | 2006-06-30 | 2009-05-19 | Dot Hill Systems Corporation | System and method for sharing SATA drives in active-active RAID controller system |
US20080005470A1 (en) * | 2006-06-30 | 2008-01-03 | Dot Hill Systems Corporation | System and method for sharing sata drives in active-active raid controller system |
US20080201616A1 (en) * | 2007-02-20 | 2008-08-21 | Dot Hill Systems Corporation | Redundant storage controller system with enhanced failure analysis capability |
US7681089B2 (en) | 2007-02-20 | 2010-03-16 | Dot Hill Systems Corporation | Redundant storage controller system with enhanced failure analysis capability |
US20160191454A1 (en) * | 2011-11-22 | 2016-06-30 | Microsoft Technology Licensing, Llc | Providing network capability over a converged interconnect fabric |
US11327858B2 (en) | 2020-08-11 | 2022-05-10 | Seagate Technology Llc | Preserving data integrity during controller failure |
US11593236B2 (en) | 2020-08-11 | 2023-02-28 | Seagate Technology Llc | Preserving data integrity during controller failures |
Also Published As
Publication number | Publication date |
---|---|
JP2003330626A (en) | 2003-11-21 |
US7480815B2 (en) | 2009-01-20 |
GB2388700B (en) | 2005-12-14 |
US20050246579A1 (en) | 2005-11-03 |
GB2388700A (en) | 2003-11-19 |
DE10317925B4 (en) | 2005-05-04 |
DE10317925A1 (en) | 2003-12-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7480815B2 (en) | Controller communications over an always-on controller interconnect | |
US7600152B2 (en) | Configuring cache memory from a storage controller | |
US6732243B2 (en) | Data mirroring using shared buses | |
US6006342A (en) | Failover and failback system for a direct access storage device | |
KR100275900B1 (en) | Method for implement divideo parity spare disk in raid sub-system | |
US7185222B2 (en) | Apparatus, system, and method for maintaining data in a storage array | |
JP3732869B2 (en) | External storage device | |
US9104790B2 (en) | Arranging data handling in a computer-implemented system in accordance with reliability ratings based on reverse predictive failure analysis in response to changes | |
US6571354B1 (en) | Method and apparatus for storage unit replacement according to array priority | |
US8370571B2 (en) | Transfer control of a storage volume between storage controllers in a cluster | |
TWI291103B (en) | Method, storage controller, system, and computer-readable recording medium for autonomic power loss recovery for a multi-cluster storage sub-system | |
US20070067666A1 (en) | Disk array system and control method thereof | |
US20040250019A1 (en) | Storage system, controller, control method and program product therefor | |
WO2017162177A1 (en) | Redundant storage system, redundant storage method and redundant storage device | |
US20110145452A1 (en) | Methods and apparatus for distribution of raid storage management over a sas domain | |
US7103717B2 (en) | Disk array device and data processing method thereof | |
WO1999063438A1 (en) | Apparatus, system and method for n-way raid controller | |
US7356728B2 (en) | Redundant cluster network | |
JP4939205B2 (en) | Apparatus and method for reconfiguring a storage array located in a data storage system | |
CN105786414A (en) | Memory system as well as access method and access device thereof | |
JP2000181887A5 (en) | ||
US8108580B1 (en) | Low latency synchronous replication using an N-way router | |
US7506201B2 (en) | System and method of repair management for RAID arrays | |
JP3776438B2 (en) | Storage device | |
JP3399398B2 (en) | Mirror Disk Recovery Method in Fault Tolerant System |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: HEWLETT-PACKARD COMPANY, COLORADO Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RUST, ROBERT A.;VAN DE GRAAFF, TAMMY T.;OLDFIELD, BARRY J.;REEL/FRAME:013131/0928 Effective date: 20020510 |
|
AS | Assignment |
Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., COLORAD Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD COMPANY;REEL/FRAME:013776/0928 Effective date: 20030131 Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P.,COLORADO Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HEWLETT-PACKARD COMPANY;REEL/FRAME:013776/0928 Effective date: 20030131 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |