TITLE OF THE INVENTION
ADAPTIVE AND SCALABLE SERVER CONTROL
CROSS REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. §119 (e) to provisional patent application serial number 60/137,354 filed June 3, 1999, the disclosure of which is hereby incorporated by reference.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
N/A
BACKGROUND OF THE INVENTION The present invention relates generally to network management systems, and more particularly to a system for managing the scalability of a system including software and hardware resources.
When a software application is installed onto a hardware platform, the amount of work (or "workload") the combined hardware and software resources can process is known as the "capacity" of the system. In order for the system to process an increased workload, its capacity must be increased, often by adding hardware resources. The potential for the system to increase its capacity, with a minimum impact on performance and availability, is referred to as its "scalability".
Users often desire to grow system capacity over time, as their business needs expand. However, existing systems frequently do not provide users with the ability to manage the scalability of their systems in an efficient and convenient fashion. Some existing systems attempt to determine the resources required for a
existing systems attempt to determine the resources required for a specific workload based on software simulations. Simulation based approaches to capacity prediction have significant drawbacks. Most fundamentally, pure simulations of real production environments are costly to develop, and are inaccurate in many cases. Accordingly, users have often found that the only practical approach to determining their system resource requirements is through trial and error. This often results in the user initially misjudging his or her specific resource requirements, and the resulting system failing to provide an acceptable level of performance.
Moreover, existing approaches to capacity planning using simulations have focused on determining how fast certain software can execute on certain hardware platforms. This approach fails to recognize that growing workload requirements generally result, at some point, in the system reaching its capacity limit. Existing systems have, therefore, not sufficiently considered how to manage the scalability of a system as the system approaches and reaches its capacity limits. In particular, network operators must often spend many hours attempting to estimate what resources are required for the network management system (NMS) itself. Such estimates are often inaccurate due to unforeseen run-time factors that have a significant impact on NMS performance. Accordingly, trial and error has been necessary in many cases to determine the actual system resources required to provide an acceptable level of capacity within the NMS itself.
For these reasons, it would be desirable to have a system for adaptively managing the scalability of a computer system, which does not require or rely on costly, time consuming simulations. The system should enable a network operator or manager to manage a network management system as its capacity limits are approached and reached. The system should further enable a network operator or manager to effectively determine the hardware resources
necessary to maintain a desired level of service within the network management system, m the face of an increasing workload.
BRIEF SUMMARY OF THE INVENTION
A system for server process control, including an administration server process which monitors the performance of a number of server processes and which is responsive to a minimum level of service that the system must meet when it accepts a request for service from a client system. The minimum level of service is referred to as a Quality of Service (QoS) . The server processes include a Network Management Server (NMS) of which the administration server is a part. The QoS of the NMS is referred to as the QoNMS. QoS or QoNMS may be specified by the user or the network operator on a per session basis, for example, within each request for service, or on a per user basis, or as a single value across all users and all sessions. When the system accepts a request for service, it may create one or more additional server processes to provide service to the requesting user. The system further shuts down one or more unneeded server processes when the demand for the particular service type associated with those server processes falls below a pre-determined level. In addition, the system monitors utilization of hardware resources that are available to support future service requests. When the amount of available hardware resources drops below a predetermined threshold, the present system provides an alarm or other notification to the network operator or user.
The disclosed server control system efficiently manages the scalability of a software system without relying on simulation results, and alerts the user when additional hardware resources are likely to be required. The disclosed system provides a predefined level of performance to each service user, automatically creates new server process instances as needed to
meet increasing service demand, and adaptively deactivates server processes as demand decreases.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING
The invention will be more fully understood by reference to the following detailed description of the invention in conjunction with the drawings, of which:
Fig. 1 is a block diagram showing the architecture of an illustrative embodiment of the disclosed system;
Fig. 2 is a block diagram showing organization of software functionality in an illustrative embodiment of the disclosed system;
Fig. 3 is a functional block diagram of an illustrative administration server;
Fig. 4 is a block diagram showing operation of a service brokerage mechanism;
Fig. 5 shows an illustrative host table;
Fig. 6 shows an illustrative service instantiation table; and
Fig. 7 is a flow chart showing steps performed during operation of the disclosed system.
DETAILED DESCRIPTION OF THE INVENTION
An illustrative embodiment of the disclosed system, as shown in Fig. 1, includes a client system 10, first server system 12, managed element 14, and second server system 16. The client system 10 is, for example, a workstation, personal computer (PC) , or other microprocessor based system, and supports execution of software including a client process 18, shown for purposes of illustration as a Java™ application within a Java Applet 20, which in turn is part of a Java Plug-in 22 executing in the context of a Web Browser 24. While the embodiments shown in Fig. 1 and elsewhere employ aspects of the Java™ language and runtime environment developed by Sun Microsystems Inc., the invention is not limited to such implementations, and other appropriate computer programming languages and software runtime environments may be used in other embodiments of the invention.
As shown in Fig. 1, the client process 18 communicates with a first server process 28 executing on the server system 12 via the Java RMI (Remote Method Invocation) protocol. The server system 12 is also for example a workstation, personal computer, or other microprocessor based system. The software executing on the server system 12 includes a server process 28, shown for purposes of illustration as a Java application executing on a Java virtual machine (VM) 30, within a Network Management Server 32. The server process 28 also communicates via the Java RMI protocol with a second NMS server 34, which similarly includes a server process executing on a Java virtual machine (VM) , and which is also shown executing on the server system 12. The server process 28 further communicates via Java RMI with a managed element object 36, shown as a Java application, executing within a run-time environment 38 such as Tornado™ for Java as provided by WindRiver Systems™, Inc., on a microprocessor based embedded system running a real-time operating system 40 such as VxWorks™, also provided by WindRiver
Systems™, Inc. The first server process 28 further communicates via the Java RMI protocol with an element simulator process 42 executing on a Java virtual machine 44 within an element simulator process 46 on the second server process 16. The embodiment of Fig. 1 provides several advantages. First, by executing the element simulator 46 in conjunction with the actual operation of other system components, more accurate estimates of system performance including the simulated element may be obtained. Second, the server to server, server to client, and server to managed element communications are all handled in a peer-to-peer fashion, for example using the Java RMI protocol. In addition, use of a virtual machine architecture allows for code produced on one platform to be executed on another platform without porting. Fig. 2 illustrates organization of software processes in an embodiment of the disclosed system. As shown in Fig. 2, an NMS Client 50 communicates with an administration server 52, which provides service brokerage and system administration services. The NMS Client 50 further communicates with services 54, which include both a Java RMI registry, as well as an HTTP server. The RMI registry stores RMI references to server processes on the local host. The HTTP server within the services 54 communicates with a Remote Service Repository 56, which stores executable code of remote server processes. The administration server 52 and services 54 operate within an NMS Server 51.
The NMS Server 51 further includes a number of managed server processes 58, database alert functionality 60, a database access interface 62, polling notification service 64, interrupt notification service 66, and network access functionality 68 for accessing the managed element in the embedded system 70. The server processes 58 are shown including servers corresponding to network management layers defined in the telecommunications managed network (TMN) developed by the International
Telecommunications Union (ITU) , including an element management layer (EML) , network management layer (NML) , service management layer (SML) , and business management layer (BML) . The database alert functionality 60 and database access interface 62 communicate with the database server 72, which stores the host table and service instantiation table, and provides the associated database services.
Fig. 3 is a functional block diagram of an illustrative embodiment of an administration server and other software components in the disclosed system. As shown in Fig. 3, a service
-user 80 communicates with a first service brokerage system 84 through a user interface 82, which is for example a graphical user interface (GUI) . Administration client software 90 communicates with the service brokerage system 84 through an administration interface 91. The administration interface 91 provides the ability to add or delete a server instance on a host (as during an upgrade to a new server release) , modify the QoS or QoNMS thresholds for the system or a specified host, obtain a list of activatable server instances on a host, obtain a list of executing server instances on a host, create or shutdown a server instance, or obtain a status report with regard to executing server instances .
The service brokerage system 84 communicates with a second administration server 86, for example using the Java RMI protocol. The service brokerage system 84 communicates with the database 88 through a database interface 93, which includes both database access and change notification functionality. The change notification functionality is capable of notifying the service brokerage system 84 when pre-specified types of changes occur in the contents of the database 88. The service brokerage system 84 also communicates with an RMI registry 94 through an RMI interface 92. Also in Fig. 3, a Java RMI activation system daemon (RMID) 98 is shown in communication with a number of managed server
instances 96, as well as with the RMI interface to the service brokerage system 84.
The managed server processes 96 are shown as FCAPS_1.0_00, FCAPS_1.0_01, FCAPS_2.0_00, FCAPS_2.0_01, in order to depict a variety of server processes within the OSI (Open Systems Interconnection) defined standard categories of Fault, Configuration, Accounting, Performance, and Security (FCAPS) management. In an illustrative naming convention, each service type is given a name, for example DBEventService for database event servers, and each server process of that type is named DBEventServicelmpl. An "incarnation number", representing the total number of server processes of the same service type that had been instantiated on the relevant host, at the time the server process was instantiated, and a release or version number of the server process are then both appended. In this naming scheme, a server process with a name of DBEventServiceImpl_2.0_00 is a database event server instantiation (as indicated by the portion of the name "DBEventService"); is the second version or release of that server process (as indicated by the "2.0" appended value); and is the first instance of that service type to be created on its host (as indicated by the "00" appended value) .
Fig. 4 illustrates operation of a transparent service brokerage system, corresponding to the service brokerage system 84 in Fig. 3. Fig. 4 shows how the present system operates to provide pointers (known as "references") to server instances on the host 110. The references provided by the present system may be used directly in the local virtual machines of the remote systems 120 and 122. Because the references are for server instances that are remote to the systems 120 and 122, the references are considered to be, and referred to as "remote references". The remote references provided by the administration server in the illustrative embodiment are implemented using the Java Remote Method Invocation (RMI) protocol, but may
alternatively be implemented using any other remote procedure call (RPC) mechanism which permits pointers to server object instances on a server system to be passed from the server system to a client system, and then permits the client system to refer to the server object instances as if they were local objects. Such an alternative embodiment may, for example, implement the Common Object Request Broker Architecture (CORBA) , as defined by the Object Management Group (OMG) .
The host 110 is shown having a Java RMI registry 112. The Java RMI registry 112 includes a number of entries which may be indexed by the administration server using • Uniform Resource
Locators (URLs), in order to obtain Java remote references to the server instances associated with the entries. Each entry within the RMI registry 112 contains an RMI reference corresponding to the URL which indexes that entry. The RMI references in the RMI registry 112 correspond to server instances within the local host
110. For example, the RMI reference 112a in the RMI registry 112 corresponds to the server instance "namel" as available on the
TCP/IP "port 1" of the host 110. The RMI reference 112b in the RMI registry 112 corresponds to the server instance "name2" as available on "port 2" of the host 110, and the RMI reference 112c in the RMI registry 112 corresponds to the server instance "name3" as available on "port 3" of the host 110.
Fig. 5 shows an illustrative embodiment of a host table, as is stored in the database 88 shown in Fig. 3, or in the database server 72 as shown in Fig. 2. The host table includes entries (142, 144 and 146) which each store information regarding a type or category of server instances on a particular host system. The information regarding a type of services on a given host includes the host IP (Internet Protocol) address 130, a ServiceName 132 indicating the service type, a QoSThreshold value 134 indicating the maximum number of server instances of the service type which can be created on the host system, a NumOfInstances value 136
indicating a current number of server instances of the service type which have been created on the host system, and a Version 138 which reflects the version of the current server instances of the service type on that host. For example, entry 142 contains information related to server instances that are of type ConfigService (configuration servers), on a host system having an IP address of 10.1.1.120. Entry 142 indicates that no more than 100 instances of such servers may be created on that host at any one time, in order to provide the Quality of Service (QoS) that has been guaranteed to all users. Entry 142 further indicates that there are currently 5 instances of Version 1.0 ConfigService servers on the 10.1.1.120 host. Similarly, entry 144 indicates that for host 10.1.1.224, there are currently 2 instances of Version 1.0 FMService (fault management) servers, out of a possible maximum 10 instances, and entry 146 contains values indicating that one instance of a version 1.0 DBAccService (database access service) server is currently on host 10.1.1.224, out of a possible maximum 100 instances of that server type that may be created on that host. Fig. 6 shows an illustrative service instantiation table, which may be stored in the database 88 as shown in Fig. 3, or within the database server 72 as shown in Fig. 2. The service instantiation table includes entries (170, 172 and 174), each of which contains information regarding the state of a server instance on a particular host. For a particular table entry, the location of the associated server instance is identified by a URL field 162, a status field value 164 indicates the current status of the server instance, a load field 166 value indicates the utilization level of the server instance, and the version of the server instance is shown by the contents of a version field 168. For example, entry 170 describes a server instance that is the 15th incarnation of that server instance type on the host on which it is executing, the location of the server instance is
RMI://10.1.1.120:1099/ConfigService_1.0_00, the server instance is currently executing (status = "UP"), the server instance is version 1.0 of that server type, and the current load value for the server instance is 50. At any given time, a server instance that has been created (or "spawned") may be in either the "Up" or "Activatable" state. A newly created process begins in the Activatable state. Upon the first reference to that instance, a fault occurs causing the Java RMID to activate the server instance. The server instance is then executing, and is considered to be in the Up state. If at a later point in time the server process is no longer "needed, it can be deactivated (or "shutdown"), which causes the server instance to release resources necessary for execution, and to transition to the "Activatable" state. The load value 166 may represent the number of user sessions with the server instance associated with the entry, the average request queue length for each session with the server instance, the rate at which requests are being processed by the server instance, an error rate associated with the instance, the time that the instance has been executing, the number of threads of execution associated with the instance, or some other suitable statistic which would be useful in examining the current utilization of the instance. The entry 172 contains field values indicating that an instance located at RMI://10.1.1.224:1099/FMService_2.0_00 is the 20th incarnation of a server of type FMService on the host system, version 2.0, that it is currently executing, and that its current utilization is reflected by a load value of 9. Finally, the entry 174 corresponds to a server process instance which is located at RMI://10.1.1.224/FMService_1.0_01, which is the 30th incarnation of the service type FMService version 1.0 on the host system, and which is currently activatable. An instance that is not "Up" is not executing, and accordingly will have a load value of zero.
As server instances are created, for example in response to configuration parameters provided by a network operator, they each register themselves in the RMI registry of their local host, as well as in the host table and service instantiation table data structures of the local administration server. More specifically, when a server process registers itself, it creates an entry in the service instantiation table as shown in Fig. 6. The URL of the server instance is stored as an RMI reference, by prepending the IP address of the host on which the server instance is located, as well as the port number of the port on that host through which the instance is reachable, to the server instance name. For example, the server instance name DBEventServiceImpl_2.0_00 could have a URL field value in the service instantiation table of RMI://10.1.1.120:1099/DBEventServiceImpl_2.0_00. This RMI reference would be used to bind a user request for a database event server to a remote reference to the specific server instance accessed through this RMI reference. In this way a JAVA remote reference is returned by the administration server to the client that made the service request. The client may then use the provided remote reference to invoke methods directly with the server instance. If the server instance is in the "activatable" state, the first such invocation will transition the server instance into the "up" state.
Fig. 7 shows steps performed during operation of the disclosed system. The present system permits a distinction to be made between services and servers . Services are exposed to the client through a unified view of a service type that may actually be provided from various remote, shared resources. The server instance that actually supplies the requested service is an implementation entity (a server or service "instance") that is hidden from the client except for a reference provided by the administration server which may be used by the client to invoke
methods on the server instance as if it were located within the local client system.
The disclosed system acts as an intermediary between service users and service providers. A service user, instead of directly looking up a service implementation in the RMI registry, sends a service request 178 to the administration server, asking for a service by specifying the type of service that is needed. For example, if the user requires a configuration server, the service request 178 would simply indicate that a server process of type "ConfigurationService" was needed. The service request 178 may also include a minimum level of service, or Quality of Service (QoS) that must be met for the request to be granted. Quality of Service may alternatively be provided by a system administrator on a system wide basis. The specific parameters which are used to indicate the minimum Quality of Server may specify one or more of the following attributes of a server instance's performance:
a. a maximum average time to process user requests b. a maximum average database read time c. a maximum user request queue depth d. a maximum average response time from the server system on which the server instance is created e. a maximum average database write time f . a maximum average depth of a communications queue within a network element on which one or more of said plurality of server instances is executing g. a maximum rate of internal communications errors h. a maximum average number of outstanding events i. a maximum event correlation queue depth
As shown in step 176 of Fig. 7, the administration server analyses the current number and utilization of "available" server instances of type ConfigurationService (those that are either
currently executing ("Up") and not fully loaded, or that are Activatable) , and the guaranteed Quality of Service (QoS) . Based on this information, the administration server determines at step 180 whether there are sufficient resources available in the system to grant the request. The administration server may also, at step 180, contact one or more candidate servers to determine whether they have sufficient available resources to provide the requested service. Alternatively or additionally, at step 180, the administration server may compare a requested minimum service level (QoS) contained within the request for service with a current system capacity, as determined at step 176, to determine if there are sufficient available resources to satisfy the request.
If there are not sufficient resources available in the system to grant the request, or if all contacted candidate systems are unavailable to provide the requested service, then step 180 is followed by step 182 at which point the request is denied. If the request can be granted, then step 180 is followed by step 184, in which the request is accepted and the administration server returns a reference to a server instance. In the illustrative embodiment, the reference returned at step 184 is a remote reference to a JAVA object that is the server instance, and that is executing on a server system that is remote from the client system from which the service request was received. The reference returned at step 184 may be any pointer with which the requesting client can subsequently directly invoke methods on the server instance, without requiring the requesting client to perform any additional steps to determine the specific location the server instance. At step 186, the administration server creates any necessary server instances of the service type indicated in the service request 178. For example, the administration server creates a new server instance in the case where all currently executing server
instances are fully utilized and the Quality of Server (QoS) related thresholds for maximum numbers of instances would not be exceeded. Subsequent invocations of the newly created server instance are performed directly by the client using the server reference provided by the administration, without intervention of the administration server.
In an illustrative embodiment, activation of server instances occurs in response to the client's first use of the remote reference provided by the administration server. Such activation is accomplished in the illustrative embodiment using the Java RMI object activation protocol, through the JAVA RMID activation daemon. Any other activation system providing persistent references to objects, as well as the ability to control object execution on an as-needed basis may alternatively be used for this purpose. Accordingly, in the illustrative embodiment, when an activatable server instance is accessed (via a method invocation) , if that server instance is not currently executing, the system initiates execution of the server instance within a JVM (Java Virtual Machine) on the server system. At step 188, the utilization of each instantiated server is determined by the administration server using information within the host table (Fig. 5) and the service instantiation table (Fig. 6) . If the average load level for a given group of server instances of a single service type, for example as indicated by the load field contents of the associated entries in the service instantiation table, falls below a predetermined level associated with that service type, then some number of those server processes are identified at step 188 as being unneeded. The number of server instances identified as unneeded is such that if each unneeded server instance is shutdown, the average load level for the remaining server instances remains below a second predetermined level. If no server instances are identified that
are unneeded, then step 188 is followed by step 192. Otherwise, step 188 is followed by step 190.
At step 190, the server instances that were determined to be unneeded in step 188 are shutdown, for example using the Java RMID (Remote Method Invocation Daemon) utility, such that their state is changed from "Up" to "Activatable" in the service instantiation table. Resources necessary for execution of each shutdown process are deallocated, and thus made available for other processes. Step 190 is followed by step 192. At step 192, the administration server determines whether the hardware resources of the system are loaded to a point where they are nearing their maximum processing capacity. This determination uses information stored within the host table and the service instantiation table. For example, if the administration server determines that the difference between the number of instances of a particular service type on a particular host, and the Quality of Service (QoS) threshold for that service type on that host, is less than a predetermined value, then the administration server notifies the NMS client or network manager at step 194. The notification may be through an electronic mail message, an alarm mechanism, or an entry in an event log that is accessible to the network manager or NMS client.
Another feature of the embodiment of the disclosed system shown in Fig. 7 provides monitoring of the number of times that a request for service is denied at step 182. As this number increases, the network operator can assume that the capacity of the system has been or will shortly be reached. Accordingly, the administration server can be configured to automatically lower the minimum service level in response to the number of times requests for service are denied reaching or exceeding a predetermined limit. In addition, the administration server will lower the minimum service level in response to an indication to do so
received from a network operator interface, such as the administration interface 91 as shown in Fig. 3.
Those skilled in the art should readily appreciate that the programs defining the functions of the present invention can be delivered to a computer in many forms; including, but not limited to: (a) information permanently stored on non-writable storage media (e.g. read only memory devices within a computer such as ROM or CD-ROM disks readable by a computer I/O attachment) ; (b) information alterably stored on writable storage media (e.g. floppy disks and hard drives) ; or (c) information conveyed to a computer through communication media for example using baseband signaling or broadband signaling techniques, including carrier wave signaling techniques, such as over computer or telephone networks via a modem. In addition, while the invention may be embodied in computer software, the functions necessary to implement the invention may alternatively be embodied in part or in whole using hardware components such as Application Specific Integrated Circuits or other hardware, or some combination of hardware components and software. While the invention is described through the above exemplary embodiments, it will be understood by those of ordinary skill in the art that modification to and variation of the illustrated embodiments may be made without departing from the inventive concepts herein disclosed. Specifically, while the preferred embodiments are disclosed with reference to certain properties and features of the Java programming language and execution environment, the present invention is generally applicable to any other programming language or execution environment which provides persistent references to objects, peer-to-peer inter-process communication capabilities, and the ability to control object execution on an as-needed basis. Moreover, while the preferred embodiments are described in connection with various illustrative data structures, one skilled in the art will recognize that the
system may be embodied using a variety of specific data structures. Accordingly, the invention should not be viewed as limited except by the scope and spirit of the appended claims.