CA2327211A1 - Management of log archival and reporting for data network security systems - Google Patents

Abstract

A system for security management comprising log archival and reporting is provided using a novel architecture with particular application which is scalable for larger scale global data networks. The system comprises a Log Collection unit, interfacing with a Data Analysis and Log Archival unit, and a Data and System Access Unit interfacing with the Data Analysis and Log Archival Unit. The Log Collection Unit comprises a Log Manager for managing log collection from a plurality log collectors interfacing with one or more security devices. The Log Manager may include a log collection manager for managing log collection from a plurality of log collectors. The log collector provides output to a log manager which provides output to a Storage Manager and a Data Analysis manager, connected to a Data Analysis Store, of the Data Analysis and Log Archival unit, which also comprise a Archival unit associated with the Storage unit. The Data and System Access unit provides a user interface, preferably web based.

Description

MANAGEMENT OF LOG ARCHIVAL AND REPORTING FOR
DATA NETTnTORK SECURITY SYSTEMS
FIELD OF THE INVENTION
This invention relates to management of security systems for data communications networks, and in particular relates to security audit logs and. management of security device log archival and reporting.
BACKGROUND OF THE INVENTION
With increasing regularity, public and private data networks are interconnecting mission critical systems. As a result, the security of these data networks has become a growing concern. Security audit logs provide a mechanism for detecting compromise of network devices by maintaining an audit trail of user activities and events generated by the various systems that make up the network.

2) Audit trails provide a :means to accomplish several security related objectives. These include, for example, individual accountability, reconstruction of past events, intrusion detection and problem analysis.
2 .'i Basic security log reporting is provided by several companies products , a . g . WebTrendsTM and TelemateT"'' which are two of the most popular firewall log analysis products currently on the market.

Nevertheless, these applications are currently limited in their ability to scalE=_ to large enterprises and global operations. There are currently no offerings available which would be suitable for large carrier class customers (ASPs Application Service Providers and Internet Service Providers ISPs).
These systems are increasingly complex, and link heterogeneous security devices, e.g. firewalls, extranet io switches and drop boxes, distributed globally.
The lack of scalablity of current systems arises from several factors.
~ Current systems archive logfiles to a general database table where logfile analysis is then done by performing select queries using the fields defined in the database table. This is not a scalable solution today, and will become even less so <~s the volumes of data increase.
~ One of the main issues in scaling current systems is that they involve a direct. connection of all security devices to the main logging/archival server" which is not compatible with globally distributed systems.
2~ ~ Security is an issue where, as in current systems, all security devices need to contact the log archival server, and it is therefore necessary to firewall the one-to-many connection with these units.

3~) Thus larger customers n~sed a system which overcomes these issues and provides features including automated summary and performance metrics, custom log analysis capabilities, and an ability to key into unusual activity and possible abuse, and trigger alarms. Other desirable :features include trend 3~~ analysis. Different levels of author ized access are required and special access capabilities for security investigations.
With respect to archival, typically security logs are treated as confidential_ corporate data, and may require that security logs are archived anywhere from a few months to a number of years.
SU1~IARY OF THE INVENTICIN
Thus the present invention seeks to circumvent or overcome the above mentioned problems by providing a novel architecture for security management systems comprising log archival and reporting for data networks, with particular application for :larger scale global data networks.
Thus, according to an aspect of the invention, there is provided a security device log and reporting system wherein archival of log files is separated from analysis of logfiles.
Separation of log file analysis and archival provides for improved scalability of the system.
According to another aspect of the invention there is provided security device log and reporting system comprising a Log Manager, the Log Manager having a distributed interface for receiving logfiles from a plurality of security devices, and is the interface to a Data Analysis and Archi,;ral unit of the system.

Beneficially the Log manager comprises an intermediary caching system for log files received from the plurality of security devices.
Advantageously, the system comprises an Data Analysis and Archival Unit, a Log Collection Unit comprising a Log Manager, and Data and :>ystem Access Unit, wherein Data Analysis and Archival Unit interfaces with only a Log Manager and a Data and System Access Unit, whereby interfaces are easily protected via a firewall and instrusion detection system.
Another aspect of the invention provides a security device log and reporting system for a data network, comprising:
a Log Collection unit, for collecting log files from security devices, a Data Analysis and Log Archival unit for analysis and archival of log files, and a Data and System Access Unit providing a user interface with the Data Analysis and Log Archival Unit.
Beneficially, the Log Collection unit comprises a Log Manager for managing log collection from a plurality of security devices.
Alternatively, the Log Collection unit comprises a plurality of log collectors and a log collection manager for managing log collection from a plurality of log collectors.
Another aspect of the invention provides a data network security management system for security device log archival and reporting comprising:

a log collection units comprising a plurality of log collectors, each for collecting log files from a plurality of security device node and a log manager for collecting log files from a plurality of log collectors;

5 a data analysis and loci archival unit for archival and automated analysis of 7_og files received from the log manager.
and a data and system access unit providing a user interface to the Data Analysis arid Log Archival Unit.
l0 The log collector provides output to a storage manager and a Data Analysis manager, connected to a Data Analysis Store, of the Data Analysis anal Log Archival unit, which also comprises a Archival unit associated with the Storage unit.
Preferably, the user interface is a web based user interface, and the data and system access unit wherein the user interface provides for log analysis summaries, trend analysis, controlled operational access and system configuration For security, the access unit comprises an authenticated, authorized, secured web based system.
The system is designed so that the log collector may receive logfiles from security devices comprising one or more device types including:
Firewalls, (raptor 4, Raptor 6, CES Checkpoint Firewall-1) Remote access services (RAS), CES (Contivity Extranet Switch), 3o SPAM (Mail shield), FTP Drop Box and Anti-Virus (Antigen) According to another a:~pect of the invention, there is provided system wherein the Log Manager LM interfaces with a Data Analysis Manages- (DAM) and a Storage Manager (SM) and the LM comprises:
means for collecting logfiles from security devices, means for pushing cached SD logfiles to a Storage manager for archival, and means for providing loci archival status updates to a Data Analysis Manager (DAM).
Another aspect of the invention provides a Log Manager for a data network security management system, wherein the Log Manager LM interfaces with a Data Analysis Manager (DAM) and a Storage Manager (SM) and the LM comprises:
means for collecting l.ogfiles from security devices, means for pushing cached SD logfiles to a Storage manager for archival, and means for providing log archival status updates to a Data Analysis Manager (DAM).
In another system according to the invention, the Log Collector Manager (LCM) interfaces with a Data Analysis Manager (DAM) and a Storage Manager (SM) and the LCM
comprises:
means for receiving logfiles from the plurality of log collectors, means for obtaining a logging system configuration from the DAM, means for propagating the configuration to individual LC
associated with Security devices, means for providing notification to the LC to begin transfer of SD log files, and means for pushing cached SD log files to the Storage manager for archival, and means for providing log archival status updates to the DAM.
According to another a:~pect of the invention the system includes a Data Analy~~is and Log Archival unit which comprises a Storage Manager (SM) and a Data Analysis Manager (DAM) and the SM compri.ses means to receive security device logs from the Log Collector 1o Manager, means for system archival, means for management of online and offline log archivals and transition of logs form online to offline status, means to provide the I:)a.ta Analysis Manager (DAM) with access to SD logs on request, and means to provide the DAM with access to the SM log Archival tables on request.
Beneficially, the system is scalable in a global environment for reasons set out below, and provides for a web interface 2o into the system.
According to yet another aspect of the invention there is provided a method of managing security device log archival and reporting for a data network security , comprising collecting log files from a security device node at a log collector collecting log files from a plurality of log collectors at a log collection manager transferring log files from the log collection 3o manager to a data analy;~is and log archival unit for archival and analysis.

Yet another aspect of t:he invention provides a method of managing security device log archival and reporting for a data network security , comprising collecting log files from a security device node at a log collector collecting log files from a plurality of log collectors at a log collection manager transferring log files from the log collection 1o manager to a data analysis and log archival unit for archival and analysis, logfile analysis being separated from log file archival.
The method may include providing user access to the Data analysis and log archival unit via a data and system access unit.
Another aspect of the invention provides a Storage Manager for a security device log archival and reporting system 2o comprising means for receiving security device logs from the log collector manager for system archival, means for management of online and offline log archival and transition of logs from online to offline status, means for providing the DAM with access to security device logs on request, means for providing the DAM with access to the SM log archival tables on request.
The storage manager benE=_ficially comprises means for 3o differentiating types o:E log files.

The following factors contribute to scalability that known logging systems do not provide.
~ The separation of logfile analysis from logfile archival.
Archival and analysis of logfiles are separated into two separate databases. The archival database (i.e. the Storage Manager), manages where logfiles are physically located on media, as well as the attributes of the logfile. The analysis of the logfile is done 1o independently of a database with only the results of the analysis stored in the analysis database (i.e. the Data Analysis Store). 'This does not preclude subsequent analyses of a logfi.le as the logfile is still available.
This approach differs from current systems which archive s5 logfiles to a general database table where logfile analysis is then done by performing select queries using the fields defined i:n the database table.
~ The Log Manager is a:rchitected and designed to be the 2o distributed interface for devices to input their logs.
The Log Manager then becomes the interface to the archival and analysis servers of the system. The Log Manager also acts as an intermediary caching system allowing end devices to offload their logs in a more 25 efficient manner. As Log Managers can be globally distributed yet still be centrally managed this increases the scalability of the system.
~ The system is architected such that the log archival and 3o analysis components are easily secured via a firewall and intrusion detection system. This i:~ achievable by the distributed component=s of the system. The only physical machines that need to interface with the archival and analysis components o.f the system are the Log Managers 35 and the Web Application Server (i.e. machine which hosts the web interface). Tnstead of having to firewall a one-to-many relationship found in current systems where all devices need to contact t:he log archival server, one only has to firewall a one-to-few relationship. The effect is 4~) that a larger, secure archival system is achievable whereas to achieve the same security with other systems it would mean managing multiple contexts of the system.
This is important in that the invention is architected and designed for the analysis and archival of 45 confidential data.

~ The ability to differentiate types of logfiles as per legal and corporate security requirements is not currently available in any other system.
5 In a system currently in operation the system handles the archival and analysis c>f 92 globally dispersed security devices on a daily basis. The device archival and analysis is provided for firewalls (Raptor 4, Raptor6, CES
Checkpoint Firewall-1) extranet switches and Remote access 10 services, SPAM (Mailshi.eld) and FTP Dropboxes and Anti-Virus. Soon to be in production will be the archival and analysis of Intrusion Detection alarms (Internet Security System's network intrusion detection), and personal firewalls (CyberArmor).
The system preferably provides authorisation support for views based on device type, database driven filter configurations and analysis store. Metrics report generation for general metrics, monthly metrics and user 2~ metrics may be generated.
Advantageously, the system uses a CORBA (Common Object Request Broker Architecture) driven system backend for communication between the system components.
2 .'i Thus the system provides for many aspects of management of log files according to corporate and la_gal requirements, automated archival and ;automated or custom analysis, and logfile exception reporting, and for example archival and 3o analysis of ISS vulnerability assessments Beneficially, such a system can be designed to be multi-vendor interoperable. Operation can be simplified if a standard format for security device lc>gs is adopted.
The systems and methods. described herein improves the ability of Security Operations to manage an increasingly complex, heterogeneous environment of Security Devices (e. g.
firewalls, extranet switches, dropboxes) through support to automation, thereby increasing the effectiveness of existing operations personnel. A level of data. security is to the infrastructure for the management of security devices protecting corporate resources and the audit capabilities of the security infrastructure are increased. The systems improves the ability t:o generate security device metrics while providing secure web access to those metrics. The resulting Security Devices Log and Reporting system provides a foundation block for an enterprise security environment called Intrusion Monitoring and Management of Unified 2~ Networks Systems {IMMUNEsystem).
Thus, the design of the system is distinguished by its architecture and functionality, in providing a system which is readily scalable for large network applications 2~> comprising globally dis~gersed security devices.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention will now be described in greater detail with reference to the attached drawings wherein:
3 () Figure 1 shows a schematic represention the IMMUNE security environment comprising a security devices log and reporting system according to an embodiment of t:he invention;
Figure 2 shows a logical view of a system according to a second embodiment of the invention ;
Figures 3 to 14 (taken from the design spec document) respectively show examples of screen prints of screen views 1o presented by the Web i_n.terface of the Web Application Server, which provides an overview of categories of screen view that may be presented to an authenticated user.
DETAILED DESCRIPTION OF THE EMBODIMENTS
[note: a glossary of acronyms presented at end of this section, preceding the Claims section) The IMMUNE security environment according to an embodiment of the invention is represented schematically in Figure 1.
2~ The system 10 for security devices log and reporting comprises the Log Collector (LC) 12, Log Manager (LM) 14, and Data Analysis and Log Archival Unit 16, and Data and System Access Unit 18. Log Collector (LC) 12 interfaces to a security device (SD) 20 which node logs events as they are processed, e.g. Firewall transactions. The Log Collector (LC) 12 transfers the security device log to be archived and analysed in IMMUNE to the Log Manager (LM) 14. The Log Manager (LM) 14 may col:Lects logs from multiple log collectors (not shown) Eor archival and analysis, and then transfers the logfiles too the Data Ana_Lysis and Log Archival Unit 16, which performs archival and automated analysis of the log files. The Data and System Access Unit 18 provides a authenticated, authorised, secured, web based access to the IMMUNE system, and provides log analysis summaries, trend analysis, controlled operations access and system configuration.
The Security Devices Log and Reporting System (SDLRS) according to a second embodiment shown in Figure 2 described below is targeted at :improving the management and access to 1o the logging of a plurality heterogeneous Security Devices (SD). for: operational and business value metrics; keying into possible abuse; legal obligations; security investigations. The SDL~RS will manage logs on a configurable basis, but the focus is on performing log analysis and log archival for SD on a daily or other regular basis.
The system according t:o the embodiment shown in Figure 2 was designed to use a known UNIX based system, for example Solaris or HP-UX for the underlying system components such 2o that the hardening of the Operating Systems adds to the overall level of system security.
Available third-party components capable of providing the intended function were used during the design phase when 2~ ever possible. The use of Internet standards-based security are utilized whenever possible.
Component Layout and Unit Description The logical view of the components making up the SDLRS 100 3o is contained in Figure :Z, which represents the system schematically, with no assumption made as to the ratio of components to computers. The server objects (e. g. Storage Manager, Log Manager) can run on the same computer or on different computers, which adds to the scalability of the solution.
There are three distinct parts of the SDLRS 100. These three parts indicated in dotted outline in Figure 2 are:
Log Collection Unit 100 Data Analysis and Log Archival Unit 200 Data and System Access Unit 300 The Log Collection Unit: 100 comprises the Log Collectors 102 which are those system modules that operate in conjunction with the logging mechanism provided by Security Devices SD
which an enterprise uses to manage data security within the s5 enterprise network The Log Collectors LC 102 interface directly with a Security device logging mechanism. The Log Collector Manager LCM 104, which provides for co-ordinating the collection of SD logs from a plurality of Log Collectors 102. The L~CM 104 transfers the logs to the Log 2o Archival Unit 200 which comprises the "Storage Manager" 202 and the LCM provides also for notifying the "Data Analysis Manager 206" of a list of newly archived SD
logs.
25 Advantageously, the Log Collector Manager LCM acts as a SD
log caching server, and the existence of the Log Collector Manager also allows for the ease of operationally securing the Data Analysis and Log Archival Unit 200 from unnecessary access by other nodes o:n the network.
The Data Analysis and Log Archival Unit comprises the Storage Manager 202 and Data Analysis Manager 206 . Storage Manager 202 which is responsible for giving the Data Analysis Manager 206 th.e identification and location of newly arrived logs, arid. for managing the archival of logs online and offline by P..rchival unit 204. Also part of the 5 Data Analysis and Log F.rchival Unit is the Data Analysis Manager 206 and Data P.nalysis Store 208. Data Analysis Manager 206 provides each system component with configuration details, analyses logs using the appropriate data filter, and sends the extracted metrics to the Data l0 Analysis Store 208. Th.e Data Analysis Store 208 is for storing system configurations, summary and operational metrics, data filter configurations, and job statuses of data analyses.

15 The Data and System Access Unit 300 cc>mprises the Web Application Server 302, Web server 304 and Web client 306.
The Web Application Server 302 includes the applications that allow the user to interface with the SDLRS for functions such as authe~ntication/access, data filters, and system setting configurations, and for the retrieval of summary metrics from the Data Analysis Store 208. As well, the Web Application Server consists of applications which allow the user to interface directly with the Data Analysis Manager 206 for applications such as custom metrics analysis, and raw data log manipulation. The Web Server 304 is responsible for providing the SDLRS screen views to the Web Client 306 for present;a.tion.
?0 In a system currently i.n operation the system handles the archival and analysis of 92 globally dispersed security devices on a daily basis. The device archival and analysis is provided for firewa.lls (Raptor 4, Raptor6, CES
Checkpoint Firewall-1) extranet switches and Remote access services, SPAM (Mailshield) and FTP Dropboxes and Anti-Virus. Soon to be in production will be the archival and analysis of Intrusion Detection alarms (Internet Security System's network intrusion detection), and personal firewalls (CyberArmor).
1o Component Detailed Description Log Collection Unit:
Log Collector (LC) A LC may be identified for each SD, or a group of SDs depending on the SD technology. In either case, the LC is responsible for the following during the log retrieval process: accessing the SD log(s), securely (i.e., authentication, privacy) transferring the SD logs) to the Log Collection Manager (LCM); cleanup of transferred logs) on the SD. As SD logging occurs as a function of the SD
software, the LC will be "tuned" to work for each type of SD. For example, retrieval of SD logs) will be on a 24 hour basis by default, but the LC will_ accept input from the LCM to increase the frequency of log retrieval in hourly intervals. Cleanup of ~>D logs will be, typicially, on a 7 day basis by default, but the LC will accept input from the LCM to increase the frequency of log cleanup in daily intervals.
Log Collector Manager (LCM) The number of LCMs in the system may be one or more with the responsibility of an LC'M being that of co-ordination and retrieval of a number of different SD operational and system performance logs. The L~CM contacts the Data Analysis Manager (DAM) on a, e.g., 24 hour basis to acquire its assigned SD
identification list, and the log retrieval and cleanup configuration settings for the system. During the log retrieval process, the LCM performs the following: initiates the 1o connection to the LC; ~>rovides system configuration updates for log retrieval and log cleanup frequencies to the LC;
securely pulls the SD l.og(s). Logs that have been securely pulled, are then securely pushed to the Storage Manager (SM) for archival with the LCM providing for each log transfer the device type, date, and SD name to the SM. Upon the transfer of an SD log(~c) to the SM, the DAM is notified of the job status, and in the case of errors the error code.
Upon completion of all log transfers, the LCM notifies the DAM with an "end of transactions" notification.
Data Analysis and Log archival Unit:
Storage Manager (SM) The SM is responsible f:or SD log archival in the correct location, maintaining an index of log archivals according to SD and export control configuration settings, and backups of the log archiving system. As part of the log transfer process, the LCM begin~> a secure log transfer to the SM with the date, device type, and SD name for the log being transferred. From thi~> information, t:he SM then selects the .o appropriate on-line archival directory where the log will be written. Upon successful completion c>f the log transfer, the SM then updates its index of log archivals.

To manage the transition of logs from on-line to off-line archival, the SM receives from the DAM the log retention configurations for the system on a daily basis. In an operational system, for' example, by default the log archival configurations are set at the following: perimeter devices -3 months on-line and 15 months off-line; export controlled devices - 3 months on-line and 57 months off-line (where a total of 60 month, or 5 year, archival. is required); drop-1o box devices - 3 months on-line and 15 months off-line;
devices classified as "other" (e. g. SF~AM logs) - 3 months on-line. Other default, values may be set as appropriate. The SM then manages the transition of on-line log archival to off-line archival by performing disk cycling, off-line archival backups, and the updating of the log archival index.
Upon receiving log location requests from the DAM, the SM
references the archival. index for the location of the log.
If the log is on-line, then the file path is given to the DAM. If the log is found to be off-line, then the DAM is informed that the log i.s off-line. Archival information for specific SD logs or for the complete on-line or off-line indices can be provided to the DAM on request.
L S
Data Analysis Manager (DAM) The DAM is responsible for providing t:he configuration details to the other system component~~, ensuring that all SD
logs are archived, perf_ormi.ng data analysis on SD logs, providing ~>ummary statistics to the Data Analysis Store (DAS), and querying the SM for log archival information upon reque:>t. To perform log analysis, the DAM runs in two concurrent capable modes:
automated analysis; custom analysis.
In the automated analysis mode, the DAM dynamically determines via DNS lookups the list of: SDs from which logs are to be retrieved. SDs having been assigned hostnames/aliases that indicate their security function and geographical location a.re then categorized into SD lists associated with the LCNf(s) in the system. The system to configuration data for log retrieval interval, log cleanup interval, log storage interval, and filter configurations are retrieved from the DAS.
When an LCM contacts t~h~e DAM, the LCM is provided with the log retrieval, and log cleanup configurations, as well as the SD list for which that LCM is responsible. The SM is notified of the system log archival configuration. The DAM
then retrieves the fi:lt.er configurations for each of the SD
categories. As the LCM(s) notify the DAM of the successful transfer of SD logs, t;he DAM then contacts the SM for the location of the SD log such that the appropriate data filter can be applied to the log. Once the data analysis is complete, the summary metrics for the SD are saved to the DAS. The DAM is respon~cible for managing the list of SD log retrievals, and the recording of errors and job statuses to the DAS.
In the custom analysis mode, the Web Application Server (WAS) contacts the DAM and requests log archival information, or the WA~~ provides the elate, time, SD
category, and name for those logs which data analysis is requested. The DAM contacts the SM for log archival information, or for the location of the log(s). In the scenario where a specific logs) are requested and found to be on-line, then the appropriate data filter configuration is retrieved from the DAS, and presented to the WP.,S for acceptance. The WAS then 5 provides the DAM with the desired data filter configuration which the DAM then applies to the SD
log(s). Summary metrics. from the custom data analysis of the logs) are then provided to the WAS.
10 Data Analysis Store (DF.S) The DAS is a database t.o which configurations, data metrics, job statuses, and authentication and access levels are stored. Configuration data exists for system archival, log retrieval., log cleanup, and the various SD
15 category data filters. Data metrics derived out of the automated analyses are stored for each SD. Job statuses and errors for all SDs are stored for each period (default is on a per day basis) of data analysis. Access and profile information for viewing system logs and configurations are a0 stored as well.
Data and System Access Unit:
Web Application Server (WAS) The WAS consists of all the applications which provide the system and data interfaces to the user. The system views consist of the following: system configuration parameters;
SD category filter configurations; access and view profile settings for definable user categories; SD and SD category data metrics views and reports; SD raw log view; alarms and job statuses.

The system configuration application presents a view for defining the system parameters: log retrieval frequency; log cleanup frequency; log archival periods; access list of certificates/ids allowed access to the system; profiles which determine what views a user has access to when authenticated. The profiles are configurable for a variable number of SD categories'., but by default the profiles are:
SECOPS (Security Operations) - access to all functionality;
SPAM (i.e. "junk e-mail") - access to SPAM log data; RAS -access to Contivity Ext.ranet switches maintained by RAS;
SECINV (Security inve~>t.igations) - access to specifiable SD
logs for security investigations.
The SD data filter application presents a view from which each SD category regular expressions can be defined for automated data analyse; and storage.
The SD data metrics applications can be either for general case metrics or custom requested metrics. In both cases, a 2o view for each SD category is presented from which the user can select the data query settings to be retrieved. The application then presents the user with the data either retrieved from the DAS (general case metrics) or from the DAM (custom requested metrics).
The alarms and statusese application presents a view which is updated with the error status and job statuses of the retrieval of SD logs. The view is dynamic in that job statuses and errors are retrieved from the DAS on an hourly ?o basis. Errors are highlighted until they have been acknowledged by an administrator. Errors and job statuses for previous dates are retrievable from the DAS.

Web Server (WS) The WS is the user's ac~~ess point into the SDLRS via the WAS. The WS authenticates the user, and sets up the SSL
connection between the WS and the user's web interface.
Web Client (WC) The WC is a web browser capable of interfacing with the WS, and hence the SDLRS.
1~
Components Inputs and Outputs This section provides further details of the component inputs and outputs used in the system according to the embodiment.
Log Collector (LC) Input from LCM:
System_configurat.ion Retrieval_Interval={default=24 hrs ~ hourly 2o interval=1 - 24 hrs}
Cleanup_Interval={ default=7 days ~ weekly interval=1 - 7days) Output to LCM:
Log transfer list:
LC Name {FQHN', IP address}
SD Name {FQHN, IP address}
Date Retrieval Interval Time Files={filet, filet, file3...) Errors file 1 file 2 file 3 Log Collector Manager (LCM) Input from DAM:
System_configurat.ion Retrieval_Interval={default=24 hrs ~ hourly interval=1 - 24 hrs}
1o Cleanup_Interval={ default=7 days ( weekly interval=1 - 7days) LC List={LC Namel, LC Name2, LC Name3...}
LC Name 'n'=~={FQHN, IP address}
SD List:={SD Namel, SD Name 2, SD Name 3...) SD_Name 'n'={FQHN, IP address}
LCM_status_request /* request status update of LC log archiving managed by LC'M */
Input from LC:
Log transfer list LC Name {FQHNf, IP address}
SD Name {FQHNf, IP address}
Date Retrieval Interval Time Files=ffilel, filet, file3...) Errors file 1 file 2 file 3 Output to SM:

Log transfer list LCM Name {FQHIV, IP address}
LC Name {FQHN, IP address}
SD Name {FQHN, IP address}
Date Retrieval Interval Time Files={filel, filet, file3...) file 1 1~ file 2 file 3 Output to DAM:
Log archival transaction complete LCM Name {FQHN, IP address}
LC Name {FQHN, IP address}
SD Name {FQHN, IP address}
Errors LCM_archival_complete /*when all logs have been 2o transferred to the SM for that interval*/
LCM_status update LC_List={LC_Namel, LC Name2, LC Name3...}
LC Name'n'={FQHN, IP address, status= [archived ~ cac:.h.ed waiting] }
Storage Manager (SM) Input from LCM:
Log transfer list LCM Name {FQHN, IP address}
LC Name {FQHnf, IP address}
SD Name f FQHI~f, IP address}
Date Retrieval Interval Time Files={filel_, filet, file3...) file 1 5 file 2 file 3 Input from DAM:
System-configuration 10 Archival_Dura.tion={typel, type2, type3...}
type'n'=:{online=[number' months], offline=[number months]}
Log Location Request SD Type 15 SD Name { FQHI~f }
Date ONLINE-OFFLInfE bit /* bit 'on' when auto analysis is being done on newly arrived logs */
Filepath List.={filepathl, fi.lepath2, filepath3...}

20 /* file path given for restored offline logs */
Log_Info Request SD Type SD Name { FQHI~f }
Date ~5 Online Table Request Offline Table Request Output to DAM:
Log Location Reply .o SD Type /* type derived from name */
SD Name {FQHN, IP address}
Date Retrieval Intf=_rval Time File Location__List={filepathl, filepath2, filepath3...}
filepath'n'=(ONLINE bit, ONLINE=filepath}
Log-Info Reply SD Type SD Name {FQHN, IP address}
LCM Name LC Name Online Offline Offline Date Online Date Log-Date Retrieval-Interval Online Table Reply Offline Table Reply Online Log Archival Table SD Type SD Name IP address LCM Name LC Name Archival Date Log Date Retrieval Interval Time={timel, time2, time3...}
Filepath={filepathl, filepath2, filepath3...}
?o Offline Log Archival Table SD Type SD Name IP address LCM Name LC Name Offline_Date Log_Date Retrieval Interval Time={time 1, time2, time3}
Filepath={N/A, N/A, N/A}
Data Analysis Manager (DAM) Input from LCM:
Log archival transaction complete LCM Name {FQHN, IP address}
LC Name {FQHN, IP address}
SD Name {FQHN, IP address}
Errors LCM_archival_complete /*when all logs have been 2o transferred to the SM for that interval*/
LCM_status update LC_List={LC_N~amel, LC Name2, LC Name3...}
LC Name'n'={FQHN, IP address, status=[archived ~ cached waiting]}
Input from WAS:
Log Location_Reque~st /* for custom analysis */
SD Type SD Name {FQHTf, IP address}
..o Date Range={Date I From_To}
Online={ONLII~fE I OFFLINE}

Offline_File_Location List={filepathl, filepath2, filepath3...}/* restored filepath known */
FULL TEXT={ON I OFF}
Custom Metrics_Request Filter Type={customized filter keys}
SD Type SD Name {FQHN}
Date Range={Date I From To}
Online Table Request 1o Offline Table Request Input from SM:
Log Location Reply SD_Type 15 SD Name {FQHN, IP address}
Date Retrieval Interval Time File Location List={filepathl, filepath2, 2o filepath3...}
filepath'n'={ONLINE bit, ONLINE=filepath}
Log_Info Reply SD Type SD Name {FQHN, IP address}
25 LCM Name LC Name Online Offlin.e Offline Date Online Date 30 Log Date Retrieval Interval Online Table Reply Offline Table Reply Input from DAS:
System Configuration Archival_Duration={typel, type2, type3...}
type'n'={online=[number months], offline=[number months]}
Retrieval_Interval={default=24 hrs I hourly interval=1 - 24 hrs}
so Cleanup_Interval={ default=7 days ~ weekly interval=1 - 7days) SDtypes={typel, type2, type3...}
type'n'={code, description}
Devicelist={devicel, device2, device3...}
Filters={fil.tertypel, filtertype2, filtertype3...}
filtert:ype°n'={keyl, key2, key3...}
Alarms={alarm.typel, alarmtype2, alarmtype3...}
alarmtype'n'={keyl, key2, key3...}
LCMlist={lcml, lcm2, lcm3...}
lcm'n'={FQHN, IPaddr, responsibility}
Output to LCM:
SD system configuration file:
Retrieval_Interval={default=24 hrs ~ hourly interval=1 - 24 hrs}
Cleanup_Interval={ default=7 days ~ weekly interval=1 - 7days) LC_List={LC Namel, LC Name2, LC Name3...}
LC Name 'n'=={FQHN, IP address}
SD List={SD Namel, SD Name 2, SD Name 3...) SD-Name 'n'={FQHN, IP address}

LCM_status_request /* request status of LC log archiving managed by L~CM */
Output to SM:
5 System Configuration Archival_Duration={typel, type2, type3...}
type'n'={online=[number months], offline=[number months]}
Log Location Request 10 SD Type SD Name {FQHN}
Date ONLINE-OFFLINE bit /* bit 'on' when auto analysis is being done on newly arrived logs */
15 Filepath List=(filepathl, filepath2, filepath3...}
Log_Info_Request SD Type SD Name {FQHN}
Date 2o Online Table Request Offline Table Request Output to WAS:
Full Text Reply 25 Logfile Text_Buffer /* for read-only access */
Custom Metrics Reply Metrics Table Status Errors 30 Alarms Search Results Online Table Reply /* summary of logs archived online */
Offline Table Rep ly /* summary of logs archived offline */
Output to DAS:
Session Analysis Date={Month, Day, Year}
Start Time 1~ Session_ID
Device Type Logfile Type Logfile Date-Time Retrieval Interval Session Results Date={Month, Day, Year}
Completion Time Session ID
Device Type Logfile Type Logfile Date_Time Error Code Alarms={none ~ [alarml, alarm2, alarm3...]}
Errors={none ~ [errorl, error2, error3...]}
Metrics={keylresults, key2results, key3results...}
key'n'results={hitl, hit2, hit3...}
Device Update Device Type Device Name 3o Status={ACTIVE, HISTORIC}
Data Analysis Store (DAS) Database Schema TABLE: analysis_session (used to store information about the logfile analysis) FIELDS:
session_id /* incorporate the date into the sessionid */
year /* Required for */
month /* ease of extraction of */
13 day /* summary metrics.*/
device_type (name of firewall contivity switch, seam machine,...) logfile_type (type of file that was parsed. ie.
some SDs will produce a number of logfiles) 15 logfile date (date and time of logfile) retrieval_interval (system log retrieval rate) start_time %* required to track DAM-system */
completion_t.ime /* performance */
TABLE: session alarms FIELDS:
session id alarmcode status /* st:atus of each alarm - active or acknowledged */
severity TABLE: session errors FIELDS:
session id errorcode status /* status of each error - active or acknowledged */
severity TABLE: logfile_types (used to store information about versions of software e.g., firewall - Raptor 4.0 vs Raptor 6.0) FIELDS:
device_type l0 logfile_type TABLE: metric_types (used to store information about the metrics that need to be calculated and where to find the results) FIELDS:
metric_id (this will be a number from 1 - 30 and is the place where the results are stored in the tables. For example, if this has a value of 2, then in the individual 2o results tables the rep>ult of this metric is stored in the metric2 field.) device_type (ie.
FIREWALL,SPAM,CONTIVITY,FTPDROPBOX,USER_STATS) logfile_type (e.g. Raptor 4, Raptor 6) metric name (this is the name that is used to describe the particular metric being found ie. Number of FTP
connects) metric key (:this is the value that is being searched ie. ftp.*connection for) status (as we are storing all metrics for many years in the database, a particular metric that was used in the past may no longer be valid but still requires a placeholder in the database for historic data. The possible entries in this field are ACTIVE, or HISTORIC where if the status is ACTIVE, then it will be used. for analysis) TABLE: user-table (used. to store information about the users accessing this tool) FIELDS:
userid (ie. CN for certs or userid) device_type (i.e.
ALL,FIREWALL,SPAM,CONTIVITY,FTPDROPBOX,USER_STATS) type_of access (e. g. DBA, ANALYST, HELPDESK, CORP-INVESTIGATIONS) user name user~hone TABLE: access (used to store information about the different levels of access) FIELDS:
type_of_access (e. g. DBA, ANALYST, HELPDESK, CORP-INVESTIGATIONS) TABLE: special_access (used to determine access rights to a log in scenarios where specific, limited access is granted) FIELDS:
userid (ie. C'N for certs or userid) device name ( i.e. ALL, FQHN(S)) /* required for security investigations; */
date (i.e. ALL, DATE RANGE) /* required for security investigatiorm: */
TABLE: firewall (used t.o store the metrics gathered on a per firewall basis per logf:ile basid - for the first cut there will be one entry per f:irewall per day but as the processing becomes more often, there may be many per firewall per day.) FIELDS:
5 session_id metric:L to metric 30 (used for counts and sums) TABLE: firewall monthly (used to store firewall information but summarized by mont:.h) 10 FIELDS:
firewall year month metricl to metric 30 15 TABLE: firewall user (used to store firewall information based on the USER_STATS) FIELDS:
transaction___type - things like connects per 20 userid, bytes transferred per userid, etc. This information is done on a per firewa.ll per logfile basis) session id userid metricl to metric 30 25 TABLE: firewall keyword'. (used to store the matched keyword information. This is done on a per firewall per logfile basis.) FIELDS:
3o session_id search key matched_line (string where the match was found) userid (if possible, the userid extracted from the matched line) count('?) (ongoing count rather than additional entries in the db?) TABLE: contivity (used to store the metrics gathered on a per contivity basis per logfile basis) FIELDS:
session id 1o metricl to metric 30 (counts and sums) TABLE: contivity monthly {used to store contivity information but summarized by month) FIELDS:
contivity year month metricl to metric 30 TABLE: contivity_user (used to store contivity information 2o based on the USER_STAZ'S) FIELDS:
transaction__type (things like connects per userid, bytes transferred per userid, etc. this information is done on a per contivity per logfile basis) session id userid metricl to metric 30 TABLE: contivity_keyword (used to store the matched keyword information. This is done on a per contivity per logfile basis.) FIELDS:
session id search_key matched_line (string where the match was found) userid (if possible, the userid extracted from the matched line) count(?) (ongoing count rather than additional entries in the db?) TABLE: dropbox (used t:o store the metrics gathered on a per so dropbox basis per logf_~ile basis) FIELDS:
session id metricl to metric 30 TABLE: dropbox monthly (used to store dropbox information but summarized by month) FIELDS:
dropbox 2o year month metricl to metric 30 TABLE: dropbox_user (used to store firewall information based on the USER_STAZ'S) FIELDS:
transaction__type - things like connects per userid, bytes transferred per userid, etc. this information is done on a per dropbox per logfile basis) 3o session_id userid metricl to metric 30 TABLE: dropbox_keyword (used to store the matched keyword information. This is done on a per firewall per logfile basis.) FIELDS:
session id keyword key (key that was looked for) matched_line (string where the match was found) userid (if possible, the userid extracted from the 1o matched line) count(?) (ongoing count rather than additional entries in the db?) TABLE: list-contivity (used to store the list of contivities that have information ~~tored in this database) FIELDS:
device_status (as we are storing metrics for many contivities for many years in the database, a particular contivity that was used in the past may no longer be valid but still requires a placeholder in the database for hi~;tori.c data. The possible entries in this field are ACTIVE, or HISTORIC where if the status is Al~'fIVE, then it will be used for analysis) ~,5 device name logfile_type TABLE: list dropboxes I;used to store the list of dropboxes that have information s>tored in this database) FIELDS:

device status (as we are storing metrics for many dropboxes for many year: in the database, a particular dropbox that was used in the past may no longer be valid but still requires a placeholder in the database for historic data. The possible entries in this field are ACTIVE, or HISTORIC where if the status is ACTIVE, then it will be used for analysis) device name 1~~ logfile_type TABLE: list_firewalls ('used to store the list of firewalls that have information stored in this database) FIELDS:
15 device-status (as we are storing metrics for many firewalls for many years in the database, a particular firewall that was used in the past may no longer be valid but still requires a placeholder in the database for historic data. The possible entries in this 20 field are ACTIVE, or HISTORIC where if the status is ACTIVE, then it will be used for analysis) device name logfile_type TABLE: list keywords (u.sed to store the list of keywords z5 that are to be used as part of an analysis) FIELDS:
search_key (search string) device_type ?0 logfile_type responsibility (group who supplied the keyword and is responsible to investigate when found - HR (Human Resources), NS (Network Security), CS
(Corporate Security)) status (as we are storing metrics for many firewalls for many years in the database, a particular firewall that was used in the past may no longer be valid but still requires a placeholder in the database for historic data. The possible entries in this field are 10 ACTIVE, or HISTORIC where if the status is ACTIVE, then. it will be used for analysis) TABLE: mailshield (used to store mailshield metrics) FIELDS:
15 session_id metricl to metric 30 (sum and counts) logfile_type TABLE: spam_rejections (used to store top 10 rejection types) FIELDS:
session id rejectl to rejectl0 occurrencel to occurrencel0 TABLE: list mailshield~. (used to store the list of mailshields that have information stored in this database) FIELDS:
device statuwc (as we are storing metrics for many .o mailshields for many years in the database, a particular mailshield that was used in the past may no longer be valid but still requires a placeholder in the database for historic data. The possible entries in this field are ACTIVE, or HISTORIC where if the status is ACTIVE, then it will be used for analysis) device name TABLE: mailshield_monthly (used to store mailshield information but summarized by month) FIELDS
mailshield year month metricl to metric 30 TABLE: blocked (used to store blocked metrics) FIELDS:
session id recipient_emailid 2o reason (store the reason that the email was blocked) subject (the subject of the blocked email) sender TABLE: owners FIELDS:
responsibility (ie, HR (Human Resources, NS
(Network Security), CS (Corporate Security)) contact name (person to contact when matched) userid contact_phone contact_email (This is key so that an email can be sent out, assuming we decide to automate this function) TABLE: error_list (used to store information about possible system errors) FIELDS:
errorno severity description 1o TABLE: alarm_list (used to store information about log alarms) FIELDS:
alarmcode severity description TABLE: device-types (used to store list of valid device_types - these will be hard-coded into this table ) FIELDS:
device-type (i.e. FIREWALL, CONTIVITY, SPAM,...) TABLE: lcm-list (used to store list of Log Collector Managers) FIELDS:
device name responsibility (string - depending on implementation could beg geographic or device type dependent) TABLE: sys_config (usecL to store list of system parameters) FIELDS:
retrieval interval cleanup_interval device_type online duration offline duration Web Application Server (WAS) WAS SCREENS:
The WAS provides a graphical user interface and Figures 2 to so 13 show some typical screen views which may be selected and which are intended to give a summary of the categories of screen views that wouldL be presented t.o an authenticated user. This summary is n.ot a complete representation of all SDLRS screen views and i.s shown by way of example.
The screen views which a user may selectare based upon the user authenticating themselves to the SDLRS, and the access rights that the SDLRS gfrants to the user upon that authentication. Depending on the authenticated user's access 2o rights, the appropriate functionality tabs at the top of each screen view will be displayed for selection.
Figure 2 represents a ~~plash Screen with Authentication:
After login and authentication by e.g. an authenticated a5 Security Operations user, the user is presented with the main menu as shown in F'igur'e 3, providing options tabs (depending on user access rights) to select Metric Results, Configure Filters, Job status , Logs Archived and Admin functions. Selection of the metric results screen as shown ..0 in Figure 4 provides options to select. results for e.g.
firewalls, contitivity switches, FTP drop boxes, SPAM, Corporate security, or return to the main menu.

The screen shown in Figure 5 represents a security devices metrics menu for firewalls. and the subsequent screen in Figure 6 shows the daily metrics screen for firewalls example.
A daily keywords results screen is shown in Figure 7 and monthly metrics screen in Figure 8.
1~ User statistics metrics screen, configure filter screen, and systems job status screen are shown in Figures 9, 10 and 11 respectively. The system logs archived screen, and system administration screen are shown in Figure 12 and 13 respectively.
WAS DATA:
Input from WS:
Authentication Request /* for logging purposes */
Userid IP Address Interactions with the DAS (Input/output):
System Configuration Archival_Dura.t.ion={typel, type2, type3...}
type'n'={online=[number months], offline=[number months]}
Retrieval_Int.erval={default=24 hrs ~ hourly interval=1 - 24 hrs}
Cleanup_Inter~val={ default=7 days ~ weekly interval=1 - 7days) SDtypes={typel, type2, type3...}
type'n'_={code, description}

Devicelist={dE=_vicel, device2, device3...} /*
Informational only as configured dynamically by the DAM */
Filters={filtf=rtypel, filtertype2, filtertype3...}
fi.ltertype'n'={keyl, key2, key3...}
Alarms={alarmtypel, alarmtype2, alarmtype3...}
alarmtype'n'=fkeyl, key2, key3...}
LCMlist={lcml, lcm2, lcm3...}
lcm'n'={:FQHN, IPaddr, responsibility}
Auth Access List 1~~ CN_List={userl, user2, user3...}
use:r'n'={access_level}
Access List=faccess_levell, access_level2, access level3...}
Session Status 15 Date Sessions=fsessionl, session2, session3...}
session'n'={status, error[1,2,3...], alarm[1,2,3...]}
error'n'={errorcode, description}
2o al.arm'n'=(description}
Metrics Reply Device Name Metricl to Metric30 25 Input from DAM:
Full Text Reply Logfile Text_Buffer /* for read-only access */
Custom Metrics Reply Metrics Table 30 Status Errors Alarms Search Results Online Table_Reply /* summary of logs archived online */
Offline Table Reply /* summary of logs archived offline */
Output to DAM:
Log Location Request /* for custom analysis */
SD Type SD Name {FQHN, IP address}
Date_Range={Date ~ From To}
Online={ONLINE I OFFLINE}
Offline_File_Location List={filepathl, filepath2, filepath3...}/* .restored filepath known */
FULL TEXT={ON I OFF}
Custom Metrics Request Filter Type={customized filter keys}
SD Type SD Name {FQHN}
Date_Range={Date ~ From To}
Online Table_Reque~st Offline Table Request Output to WS:
a5 Data fills for presentation Web Server (WS) Authenticates and establishes secure connection Presentation of system to end user Detailed design information for embodiment comprising a Log Manager (LM) In the embodiment described above, the Log collection unit comprises distinct Log Collector Manager (LCM) and Log Collector (LC) components, which are described in further detail in the LCM Design section and LC Design section following .
In the embodiment shown schematically in Figure 1, the the log collection unit comprises a Log Manager (LM), this 1o component of the Security Devices Log Reporting System (SDLRS), which is responsible for the collecting of Security Device (SD) operational logs, and the transferring of those logs to the Storage Manager (SM) for archival. In fulfilling this role, t:he LM also has a corresponding 7.5 interaction with the Data Analysis Manager (DAM) component of the SDLRS.
The intent of this section is to provide the architecture and design of the LM arid not the implementation specifics of the LM. For the ease of understanding the LM system, <';0 configuration files and tables detailed in the design, as well as example content, and records are provided to highlight key fields and information that are required by the LM. The actual implementation of the files and table content may vary.
a:5 The log manager functions to 1 provide a collection point for Security Devices (SD) to transfer their logf:iles for archival.
2 Push the cached SD logfiles to the Storage Manager (SM) for archival.
;SO 3 Log archival status updates provided to the DAM.

CORBA Integration The Log Manager (LM) acts as a CORBA client. The CORBA
server interfaces with which the LM interacts during service requests are defined in the CORBA integration document, and are referred to whenever possible. They will appear as the actual interface method name preceeded by the server entity.
For example, the notifi~~ation represented by the LM sending the DAM a Log Archival Complete notification is DAM-LogArchDone.
1~ System variables For UNIX-based LM implementations the system variables are analogous to the UNIX shell environment variables (e. g.
setenv in the csh) and can therefore be used for that purpose (e. g. setenv LMDIR <DirLoc>, for the csh) 15 CACHEDIR . DirLoc The CACHEDIR variable defines the location of the logfile cache directory .for the Security Devices) (SD). The directory contains the logfiles of SD to be transferred to the Storage Manager (~~M:). This variable symbol is also used 20 as a production in syntax definitions in this document.
LMDIR . Di.rLoc The LMDIR variable defines the location of the LM run-time directory, which contains the configuration files, Security Device File (SDF), Log Transfer List (LTL), and Log 25 Exception List (LEL) for the LM. This variable symbol is also used as a production in syntax definitions in this document.
CheckInterval The CheckInterval variable defines the number of minutes 3o between each check by t:he LM for new SD logfiles.
CleanupInterval The CleanupInterval variable defines the number of days archived SD logfiles are kept by the LM. By default the number of days equals ..5 RetrievalInterval The RetrievalInterval variable defines the number of hours an archived SD logfile will span. By default the number of hours equals 24 (i.e. 1 retrieval per day).
Configuration repository The LM configuration repository at version 1.0 will be a configuration file. It is located on the LM host and uses the following syntax:
LMConfigRep . LMD~IR "/" "LM.ini"
In future versions of ~;DLRS, the LM configuration repository so may also be available vi.a a database table. If the LM
configuration repository is a database table, then it will use the following syntax:
LMConfigRep . "LM:Config" The LM validates that the $CACHEDIR directory exists.
1) The LM validates that the Security Device File , Log Transfer List, and the Log Exception List exists.
2) The LM check: the pending activity file to see if it has any pending actions to execute or restart.
Log collection Management The LM is responsible f:or transferring Security Device (SD) logfiles to the Storage>. Manager (SM) for archival. To perform this role within SDLRS, the LM must manage the a5 following aspects of the archival process:
~ act as a temporary cache for logfiles in-transit for archival on the SNt.
~ transfer newly arrived SD logfiles to the SM for archival.
..o ~ notify the Data Analysis Manager (DAM) of SD logfiles that have been archived.
~ maintain an exception list of SD that have not submitted logfilE=_~> for archival.
Log transfer Management:

The LM manages the transfer of logfiles to the SM using a Security Device File (S:DF) and a Log Transfer List (LTL).
Security Device File The Security Device Fife (SDF) is a manually edited configuration file in $:LMDIR, and it contains information relevant to the archiving of SD logfiles, as well as in the data analysis of those logfiles. Each line in the file contains the following keys in order delimited by "white space":
10 ~ SD Name // the FQHN, e.g.
bcarh001.ca.nortel.com ~ SD Alias // e.g. fw-1-a-cc ~ SD Type // e.g. FW, SPM, etc.
~ SD_SubType // e.g. EAGLE, RAPTOR4, etc.
15 An example line is provided below:
bcarh001.ca.nort;el.com fw-1-a-cc FW EAGLE
Log Transfer List A Log Transfer List (LTL) is used to keep state of which SD
logfiles require transferring to the Storage Manager for 20 archival. Each entry in the LTL is a record consisting of the following fields in. order, and delimited in this example by two asterisks:
SD Name SD Alias 25 ~ SD Type SD_Subtype Date Interval._Stamp Retrieval Interval ..0 ~ Log_Size~ // expressed in kilobits Compres~sed_Flag Data Type Filepath // to be prepended by $CACHEDIR
An example LTL record for a firewall is given below:
bcarh001.ca.nortel.com**fw-1-a-cc**FW**EAGLE**19990910**00**24**895**y**ASCII**transfer/
bcarh001/19990910-00/$LOGFILE
Log Exception List A Log Exception List (LEL) is used to keep state of which SD have submitted logfiles for archival during the logfile retrieval interval. Each entry in the LEL is a record consisting of the following fields in order, and delimited in this example by twc> asterisks:
Date ~ Interval__Stamp SD Name An example LEL record for a SD is given below:
19990910**00**bcarh001.ca.nortel.corn Log Manager Interaction with a Security Device The Log Manager (LM) i~; an independent. process working in conjunction with third-party Security Devices (SD) for the purposes of archiving t:he SD logfiles in a managed, centralized location.
The Security Device (SL)) software must. be configured such ~5 that the following occurs on a daily basis .
~ The SD logfile(s) must be transferred via an SD
administrative process to the appropriate directory on the LM. An example UNIX directory representation is provided . LMName:$CACHEDIR/newlc>gs/$SDNAME
_,0 /$DATE.$logfile CACHEDIF; is set in the LC. in:i file SDNAME is a sub-directory created in $CACHEDIR/newlogs by the SD administrative process, which identifies the specific SD
that created the logfiles.
$DATE.$logfile . $DATE"."$logfile where:
$DATE . the date specified in YYYYMMDD
ASCII format $logfile .~ the name of the logfile generated by the SD
Preparing Security Device Logfiles for Archival 1o The LM cache directory (i.e. $CACHEDIR) contains three directories: "newlogs"; "transfer"; "archived". Newly arrived logfiles from the SDs are found in the "newlogs"
directory under the appropriate SDName directory. These logfiles must be prepared for transfer to the SM for archival. The "transfer" directory contains new SD logfiles which have been procec>sed by the LM anal are designated to be transferred to the Storage Manager (SM) for archival. The "archived" directory contains SD logfiles that have been transferred to the SM, and that are cached for the period of time specified by the $CleanupInterval.
At a regular interval determined by the value of $CheckInterval, the LM checks the $CAC'HEDIR/newlogs directory for any newly created directories. When a new directory is found, they logfiles contained in it are ~5 processed as follows:
~ Using the $DATE obtained from the logfile name (i.e.
$DATE.$LOGFILE), and the corresponding $RetrievalInterval. (e.g. 24 hrs.) for creating an IntervalStamp, the directory ..0 $CACHEDIR/transfer/$SDNAME/$DATE-$IntervalStamp is created.
An example subdirectory created in $CACHEDIR/transfer is provided below ~ $CACHEI7IR = / sdlrs / 1_ogarchive $SDNAME - bcarh001.
$DATE - 19990910 $IntervalStamp = 00 // 24 divided by 24 - 1 = "begin at midnight" - 00 ~ directory =
/sdlrs/1.ogarchive/transfer/bcarh001/19990910-~ The logfiles contained in the $SDNAME directory are then compressed (i_f not already compressed) and moved from $CACHEDIR/newlogs/$SDNAME/$DATE.$LOGFILE to the correct "transfer"' directory $CACHEDIR/transfer/$SDNAME/$DATE-$IntervalStamp/$LOCiFILE
~ A record entry for each logfile to be transferred to ~.5 the SM via the Net:File Put method is then created in the Log Transfer List (LTL). (The NetFile methods are detailed in the CORBA integration document [MA1].) Transfer of Logfiles for Archival Immediately after a period of preparing any newly arrived SD
logfiles for transfer t:o the SM for archival, the LM then transfers the logfiles associated with the entries in the Log Transfer List (LTL) to the SM using the NetFile Put method detailed in the SDLRS CORBA integration document [MA1]. Upon successfu7_ completion of the logfile transfer, ~;5 the following events occur:
~ A DAM-LogArchDone notification is sent to the DAM
indicating that the SD logfiles are ready for analysis.
move the logfile f=rom the "transfer" sub-directory $CACHEDIR/transfer/$SDNAME/$DATE--;~0 $IntervalStamp/$LOGFILE to the "archived" sub-directory $CACHEDIR/archived/$SDNAME/$DATE--$IntervalStamp/ $LOGFILE
~ remove the corres~>ondi_ng record from the LTL
~ remove the corresponding record from the LEL
.c5 Clean up of Logfiles after Archival The LM keeps the SD loc~files that have been transferred to the SM for the duration specified in $CleanupInterval. On a daily basis, the LM removes any logfiles in the "archived"
directory by doing the following:
~ using the file cre<~tion date stamp of the directory $CACHEDIR/archived/$SDNAME/$DATE-$IntervalStamp as the logfile origin date, remove any directory (i.e.
$CACHEDIR/archived/$SDNAME/$DATE-$IntervalStamp) and its contents that..have exceeded the $CleanupInterval in duration. This allows older logfiles which have been newly submitted to the LM to be archived for the 1~ desired duration.
Generating Logfile Archival Exceptions The LM keeps state of the SD which have submitted logfiles 15 to it during a $Retrievalnterval period. At the beginning of each retrieval interval period, the LM performs the following tasks in order:
~ Each record in the LEL represents a SD which did not submit its logfil.e(s) for an earlier interval period.
20 The DAM is sent a notification for each LEL record indicating that it has not received the logfile(s) for the SD during the interval specified in the LEL record.
This notificatloTl is done via DAM-Event as documented in the CORBA integration document [MAla.
25 ~ The LM appends t<:> the LEL an LEL record for each SD
listed in the Security Device File (SDF).
Concurrent Event Handling There are no special requirements for concurrency on the LM.
Activity Status file 30 The Activity Status File (ASF) contains state information for various activities going on in the LM. For example, as each logfile transfer operation to the SM is initiated, the LM stores the event related information so that if the system crashes, it can restart any pending activity.
..5 The pending activity file syntax is:
StatFile . LMDIR" ! "' "LM. stt"
The syntax for each record is:

ASFEntry . JobNumber Activity JobNumber . Integer[4J

Activity . Log Prep I Log Transfer Archival Notification ~ Cleanup Log Prep . Status ";" DateTime ";" SDName Log Transfer . Status ";" DateTime ";"

LCMName ";" SDNa~me ";" \

LogAttributes";" ErrorStatus Archival Notification . Status ";"
DateTime ";"

SDName ";" LCName";" \

LogRefs Cleanup . Status ";" DateTime Status .. "n" // :new but not acted on "s" // started job "c" // complete, just cleaning up "f" // failed, just cleaning up "r" // system failure, job restarted DateTime . hh:mm:ss // UNIX date/time (i.e. time()) Log Manager Event Logging The LM logging uses syslog. Syslog should be setup with the following parameters:
void openlog(con,st char *ident = "LM", int logopt =
LOG_PID+LOG NO~nIAIT, int facility = LOG USER);
5 A message will be issued when the following occurs:
1) LM starts up, including command line parameters 2) LM shuts down.
3) LM transfers log to SM using Ne tFile:Put method, including parameters l0 4) LM calls DAM -LogArchDone (log archival notification), including parameters 5) Significant state changes during log transfers (e. g. start, end, misc.) 6) Significant :Mate changes during the creation of 15 the Log Tran~cfer List 7) LM calls DAM -Event during archival exception notifications 8) Security related events 9) When an error occurs As much as possible the message part of the syslog() call should be in a machine parsable form.

Log Collector Manager This section contains more detailed design information for the Log Collector Manager (LCM), which is a component of the Security Devices Log Reporting System (SDLRS) of the second embodiment. The LCM i.s responsible for the co-ordination and retrieval of a number of Security Device (SD) operational logs, and the transfering of those logs to the Storage Manager (SM) f.or archival. In fulfilling this role, the LCM also has corresponding interactions with the Data io Analysis Manager (DAM) and Log Collector (LC) components of the SDLRS.
Design Representation The intent of this section is to provide the architecture 15 and design of the LCM and not the implementation specifics of the LCM. For ease of understanding the LCM system configuration, files and tables detailed in the design, example content and records are provided to highlight key fields and information that are required by the LCM. The 2o actual implementation of the files and table content may vary.
Overview Major Functions Obtains the logging system configuration from the Data 25 Analysis Manager (DAM) and propogates the configuration to the Log Collectors (LC:) corresponding to the Security Devices (SD).
Notifies the LC to begin transferring the SD logfiles.
Pushes the cached SD logfiles to the Storage Manager (SM) 3o for archival.
Log archival status updates provided to the DAM.
CORBA Integration The Log Collector Manager (LCM) acts as both a CORBA client and a CORBA server. The service requests that are defined in the CORBA integration document are referred to in this document whenever possible. They will appear as SR-n (where n is an integer) and preceded by the entity LCM. For example, the service request represented by the LCM
receiving logging system configurations from the DAM is LCM
SR-4.
Service Request Function Prototype 1o The service request functions are based on the service requests defined in the LCM entity interface in the SDLRS
CORBA integration document System Variables For UNIX-based LCM implementations the system variables are 15 analogous to the UNIX shell environment variables (e. g.
setenv in the csh) and can therefore be used for that purpose (e. g. setenv LCMDIR <DirLoc>, for the csh) CACHEDIR . DirLoc The CACHEDIR variable defines the location of the logfile 2o cache directory, which contains the logfiles of security devices in transit to the Storage Manager (SM). This variable symbol is also used as a production in syntax definitions in this document.
LCMDIR . DirLoc 25 The LCMDIR variable defines the location of the LCM run-time directory, which contains the: configuration files; Log Collector Table; Security Device Table. This variable symbol is also used as a production in syntax definitions in this document.
30 Configuration Repository The LCM configuration repository at version 1.0 will be a configuration file. It is located on the LCM host and uses the following syntax:
LCMConfigRep . LCMDIR "/" "LCM.ini"
In future versions of S:DLRS, the LCM configuration repository may also be available via a database table. If the LCM configuration repository is a database table, then it will use the following syntax:
LCMConfigRep . "LCI~lConfig"
Initialization The LCM queries the Data Analysis Manager (DAM) for its Security Device (SD) list, and the log retrieval and cleanup interval configurations for the different device types.
The LCM validates that the Log Collector Table (LCT) exists, and is populated with t:he LC list received from the DAM.
The LCM validates that. the Security Device Table (SDT) exists, and is populated with the corresponding SD to LC
data.
The LCM notifies the Log Collectors (LC) of the log 2~ retrieval and cleanup interval configurations.
The LCM checks the pending activity file to see if it has any pending actions to execute or restart.
Log Collection Management The LCM is responsible for retrieving Security Device (SD) 2~ logfiles from their associated Log Collectors (LC) and then sending them to the Storage Manager (SM) for archival. To perform this role within SDLRS, the LC:M must manage the following aspects of the archival process:
manage a dynamic list of SD that could potentially change on 3o a daily basis.
provide the LC, for which the LCM is responsible, with the retrieval and cleanup intervals.

notify the LC, for which the LCM is responsible, to begin logfile transfers for the SD associated with the LC.
act as a temporary cache=_ for logfiles :in-transit for archival on the SM.
5 notify the Data Analysi:~ Manager (DAM) of SD logfiles that have been archived.
notify the DAM that all SD logfiles associated with the LC
list have been archived.
Log Collector and Security Device Associations to A Log Collector (LC) manages the log archival for one or more Security Devices (SD) depending on the SD architecture.
For example, there is a one-to-one relationship between LC
and SD for Raptor firewalls, but there can be a one-to-many relationship between LC and Contivity Extranet Switches 15 (CES), since a LC cannot be co-located with a CES at the time of writing this document. Therefore given this relationship of possiblE=_ one-to-many SD to a LC, the LCM
must manage which LC is responsible for which SD.
Log Collector List 20 The Log Collector (LC) List is the association of SD to LC
generated by the Data Analysis Manager (DAM). From this LC
List, the LCM manages the transition of SD logfiles to the Storage Manager (SM) fo:r archival.
Acquiring the Log Collector List 2p On a daily basis, the LCM contacts the DAM for the list of LC for which the LCM is responsible for the day's log collection. Since the list of LC for which a LCM is responsible is of a potential dynamic :nature, the LCM
manages each day's LC list in a separate Log Collector Table 3~) and Security Device Table. The notification that the LCM
sends the DAM to acquire the LC list is detailed in the DAM

SR-4 (Obtain LC List) .in the SDLRS CORBA integration document [MA1].
Log Collector Management Tables The LCM manages the LC to SD relationship using two tables:
Log Collector Table (LCT); Security Device Table (SDT).
These tables are created using the data contained within the LC List. At the time t=hat the LC Lists is retrieved from the DAM, the following events occur:
the LCM checks for a valid LCT and SDT , and if they exist ..o writes the contents of the LCT and SDT to syslog as an error. The tables are then renamed with "WARNING" prepended to the table name.
the LCM creates a new LCT.
the SDT is created as t;he LCM sends "logfile transfer begin"
~.5 notifications to the LC, and receives back the expected number of intervals of SD l.ogfiles that will be archived.
Log Collector Table A Log Collector Table I;LCT) is used to maintain the status of . LC system configuration notifications; LC logfile transfer notifications; SD archival complete; LC archival complete.
Log Collector Table Naming Convention The LCT name syntax i~ as follows:
25 LCTab.Date .~- "LCTab."Date Date . MMDDYYYY' MM . (01~02~03~04~05I06~07~08~09~10~11~12) DD
(01~02~03~04~05~06~07~08~09I10~11.~[...]~20~21~[...]~30~31) 3o YYYY . Year expressed in string format Log Collector Table Format The first keys in LCT define the characteristics of 7 the the table. These tablecharacteristics are as follows:

LCT Date // Date of the LCT creation LC Count // Number of LC to manage SD Count // I~Tumber of SD with logfiles to archive LC Config Notification Count // Number of LC provided with system configuration SD Logfile Transfer Count // Number of SD begin-to transfer-notifications SD Archival Complete Count // Number of SD with logfiles successfully archived LC Archival Complete Count // Number of LC complete Each subsequent entry in the LCT is a record consisting of 1~ the following fields in order, and delimited by two asterisks (i.e. '**'):
LC Complete_Flag Config_Sent_Flag LC Name 20 LC IP Address "SD1" (", ""SD2") [...] // list of SD managed by the LC
"Log Transfer_Beginl"(",""Log Transfer Begin2")[...] //
flags for SD list 25 "Archival Completel"(",""Archival Complete2")[...] // flags for SD list where:
SD"n" . {SD_Name, SD~_IP Address}
An example LCT record is given below 30 n**y**fw-1-a-cc**47.150.48.2**bcarh001,47.150.48.2**y**n The example LCT .record indicates .
LC is still active System configuration ha.s been sent to the LC
LC name LC IP address SD Name, SD IP address Request to begin log transfer for SD Name has been sent to the LC
Archival notification t.o the DAM has not been sent Security Device Table A Security Device Table (SDT) is used to maintain the status of . logfile transfer start time; logfile transfer current time; number of logfile transfer sessions expected; number of logfile transfer sessions completed; SD logfile attributes Security Device Table Naming Convention The SDT name syntax is as follows:
SDTab.Date .~ "SDTab."Date Date . MMDDYYYY
MM . (01I02~03~04~05~06~07~08~09~10~11~12) DD
(01~02~03~04I05I06~07~08~09~10~11~ [...] I20I21~ [...] ~30~31) YYYY . Year expressed in string format Security Device Table Format The first 5 keys in the SDT define the characteristics of the table. These table characteristics are as follows:
SDT Date // Date of the SDT creation Logfile Transfer Start. Time // Timestamp - first logfile transfer completed Logfile Transfer Current Time // Timestamp - last logfile transfer completed Logfile Transfer Session Count // Number of logfile transfer sessions expected Logfile Transfer Complete Count // Number of logfile transfer sessions completed Each subsequent entry in the SDT is a record consisting of the following fields :in order, and delimited by two asterisks (i.e. '**'):
LC Name 7.0 LC IP Address SD Name SD IP Address SD Type Logfile Date Retrieval Interval "Logfile Type 1"(",""Lagfile Type 2")[...]
"Logfile Time 1" (", ""Logfile Time 2") [...]
LogCacheDir The LogCacheDir is unique for each entry in the table, and is the cache location within the $CACHEDIR for a security device's logfiles on that day. The format of the LogCacheDir is provided below .
LogCacheDir . Logfile Date/SD Name/
The logfiles within LogCacheDir reflect the Logfile Type and Logfile Time in the following format .
Logfile Type1"-"Logfile Time1"-"log An example SDT record for a firewall is given below:
fw-1-a-cc**47.150.48.2**bcarh001**47.150.48.2**fw**
19990804**24**raptor4**00**19990804/bcarh001 An example of the logfile within the LogCacheDir is given below:
19990804/bcarh001/raptor4-00-log Log Collector System Configurations The LCM is responsible :Eor retrieving from the DAM the system configurations r.=levant to Log Collectors (LC) for 5 the Security Devices (SD), and pushing these system configurations to the LC assigned to the LCM for that particular day.
Obtaining Configurations from the Data Analysis Manager On a daily basis the LChI sends a notification to the DAM to 1~) acquire the SDLRS configuration settings for retrieval and cleanup intervals, which the LCM then stores in a file in the LCM run-time directory. This notification is detailed in the DAM SR-5 (Obtain Configuration Information) in the SDLRS CORBA integration document [MA1].
15 Pushing Configurations to the Log Collectors The LCM pushes the retrieval and cleanup interval configurations to each Log Collector (LC) with an entry in that day's Log Collector Table (LCT). The configuration notification is detailed in the LC SR-2 (Set Configuration 20 Information) in the SDLRS CORBA integration document [MA1].
Transfer of Logfiles for Archival Transferring Logfiles from the Log Collectors The LCM notifies the LC to begin transferring logs to the 25 LCM through the LC SR-~3 (Transfer Log Information) notification detailed in the SDLRS CORBA integration document [MA1]. The LC' returns to the LCM the number of interval periods (e. g. default interval period equals 1 day) of SD logs that the LC intends to transfer to the LCM. The 30 logfile dates) associated with the interval periods) is passed as part of the ~>arameter list. The LCM upon receiving the intended number of logfile retrieval intervals for a SD creates a Security Device Table entry for each retrieval interval with the corresponding date associated with the retrieval interval.
After the return of the LC SR-3 notification, the LCM can expect the transferring of logfiles from the LC via LCM SR-2 (Transfer Log to LCM) f:or each corresponding interval period. An example i~; provided below:
interval period = 24 hr's = 1 day number of interval periods of SD logs to transfer = 3 days to number of logfiles per interval period for this SD = 2 logfiles LCM SR-2 called for: da.yl; day2; day3 number of logfiles transferred in each LCM SR-2 call = 2 logfiles When a LCM SR-2 (Transfer Log to LCM) is initiated, the corresponding SD logfil.es are cached in the $CACHEDIR (See the "Security Device Table Format" section for cached logfile naming convent;ions.) At the successful completion of LCM SR-2, the LCM updates the appropriate SD record in the SDT for the corresponding interval period.
Transferring Logfiles t.o the Storage Manager As the LCM receives logfiles from a LCM SR-2. call they are stored in the appropriate directory in $CACHEDIR. Once the transaction is complete and the SDT table updated, the logfiles are then transferred to the SM using SM SR-2 (Transfer Log to SM) detailed in the SDLRS CORBA integration document [MA1]. Upon successful completion of SM SR-2, the following events occur:
3o Security Device Table (SDT) characteristics are updated, and the corresponding SD entry in the SDT removed.

A DAM SR-1 (Log Archival Complete) notification is sent to the DAM indicating that the SD logfiles are ready for analysis.
The Log Collector Table (LCT) characteristics are updated, and the corresponding LC entry in the LCT updated.
If LCM log archival is :now complete for all SD, then a DAM
SR-1 (Log Archival Complete) notification is sent to the DAM
indicating that the LCM has completed all logfile archivals, and the day's LCT and S:DT are removed after writing the l0 table characteristics to the system log.
Logfile Archival Notifi~~ations to the Data Analysis Manager The LCM sends logfile archival notifications to the DAM in the case where a SD logfiles have been successfully archived to the SM, and in the case where all the SD assigned to a 1~ LCM have successfully had their logfiles transferred to the SM for archival.
Notification of Security Device Log Archival Complete Once the logfiles associated with an SD for a particular interval period have been transferred to the SM for 20 archival, the LCM sends an archival complete notification to the DAM. This notification is detailed in the DAM SR-1 (Log Archival Complete) in the SDLRS CORBA integration document [MA1]. The effect of the notification is for the DAM to begin the data analysis of the SD logfiles. For more 25 information see the SDLRS DAM design document [MA2].
Notification of Log Archivals Complete Once all of the SDs designated to the LCM by the DAM have had their logfiles archived on the SM, the LCM sends an archival complete notification to the DAM. This 30 notification is detailed in the DAM SR-1 (Log Archival Complete) in the SDLRS CORBA integration document [MA1].

The effect of the notification is to inform the DAM that the LCM has completed log a:rchivals for that interval period.
Concurrent Event Handling The nature of the LCM is that it will not have to deal with a large number of transactions-per-second (tps), but rather that the majority of LCI~ transactions will be of a long-lasting nature due to event-caused, prolonged, disk-related activity. Given these system specifics, the LCM must be able to handle multiple concurrent events. For example, a 1~ "transfer log to LCM" notification from a LC (LCM SR-2) can arrive from a LC at the same time as another LCM SR-2 is received from a different LC. Each of these events could potentially result in substantial disk activity given that logfiles can be of substantial size.
The most efficient means of handling concurrency in this scenario is through lightweight threads. In the worst case of the LCM running on a single processor system, the overhead involved in thread creation and in context switching between threads is minimal when compared to the latency times associated with disk accesses. In the best case, of multiple disk controllers, and multiple processors on a SMP (symmetrical mufti-processing) LCM system, threads would be able to concurrently process on different processors/disk controllers. For these reasons, the LCM
should be implemented using threads rather than by an event loop.
Activity Status File The Activity Status File (ASF) contains state information for various activities going on in the LCM. For example, as each logfile transfer notification from a LC is received, the LCM stores the event related information so that if the system crashes,it can :restart any pending activity.

The informationin the stat file can be displayed via LCM

SR-1.

The pending ivity file syntax is:
act StatFile . LCMDIR"/" "LCM.stt"

The syntax for each record .is:

ASFEntry .- JoblVumber Activity JobNumber . Integer[4]

1o Activity . Sys Config I Cache I SM Transfer Archival Notification Sys Config . Status";" DateTime ";" LCName ";"

ConfigInfo Cache . Status ";" DateTime ";" SDName ";"

1~ LCName ";" \

LogRefs SM Transfer . Status ";" DateTime ";" SDName ";"

LCName ";" \

LogRefs 2) Archival Notification . Status ";" DateTime ";" SDName ";" LCName ";" \

"-" LogRefID ;" ErrorStatus "

Status . "n" // new but not acted on "s" // started job 25 ~ "c" // complete, just cleaning up "f" // failed, just cleaning up "r" // system failure, job restarted DateTime . Ba.sel6 // UNIX date/time (i.e. time()) 3o in base 16 Basel6 Table The table for the basel6 representation is:
Basel6Table . "a" // for 0 "b" // for 1 "c // for 2 "d" // for 3 "e" // for 4 // for 5 "g" // for 6 1~ ( "h" // for 7 "I" // for 8 "j" // for 9 "k" // for 10 "1" // for 11 15 I "m" // for 12 "n" // for 13 "o" // for 14 "p" / / for 15 Log Collector Manager Event Logging 20 The LCM logging uses syslog. Syslog should be setup with the following parameters:
void openlog(const char *ident = "LCM", int logopt =
LOG PID+LOG NOWAIT, int facility = LOG USER);
A message will be issued when the following occurs:
LCM starts up, including command line parameters LCM shuts down LCM receives LCM SR-1 (DAM requesting status info), 3o including SR parameters.
LCM receives LCM SR-2 (caching of log from LC), including SR
parameters LCM receives LCM SR-3 (LC requesting configuration info), including SR parameters LCM receives LCM SR-4 (set configuraton information), including SR parameters LCM calls DAM SR--1 (log archival notification), including SR
parameters LCM calls DAM SR-4 (obt,ain LC list), including SR parameters LCM calls DAM SR-5 (obtain system configuration info), including SR parameters LCM calls LC SR-1 (obtain LC status), including SR
parameters LCM calls LC SR-2 (set configuration info), including SR
parameters LCM calls LC SR-3 (transfer log info), including SR
parameters LCM calls SM SR-2 (log transter to SM), including SR
parameters Significant state changes during log transfers (e. g. start, end, misc.) 2o Significant state changes during Log Significant state changes during the creation of Log Collector Management Tables Security related event;
When an error occurs a5 As much as possible the message part of the syslog() call should be in a machine parsable form.
For Future Study A tool to enter and modify LCM configuration information.
The possibility of the LCM specifying date ranges of :30 logfiles to be transferred from the Log Collectors.

Additional description of operation of the Log Collector Manager (LCM) The number of LCMs in the system may be one or more with the responsibility of an I:~CM being that of co-ordination and retrieval of a number of different SD operational and system performance logs. The L~CM contacts the Data Analysis Manager (DAM) on a 24 hour basis to acquire its assigned SD
identification list, and the log retrieval and cleanup configuration settings for the system. During the log retrieval process, the LCM performs the following: initiates the connection to the L~C; provides system configuration updates for log retrieval and log cleanup frequencies to the LC; securely pulls the SD log(s). Log" that have been securely pulled, are then securely pushed to the Storage Manager (SM) for archival with the LCM providing for each log transfer the devices type, date, and SD name to the SM.
Upon the transfer of an SD logs) to the SM, the DAM is notified of the job status, and in the case of errors the error code. Upon completion of all log transfers, the LCM
ao notifies the DAM with an "end of tran;>actions" notification.
The following lists references to the LCM in other processes taken from the SDLRS design description above.
A LC may be identified for each SD, or a group of SDs ~5 depending on the SD technology. In either case, the LC is responsible for the fol.lowi.ng during the log retrieval process: accessing the SD log(s), securely (i.e., authentication, privacy) transferring the SD logs) to the Log Collection Manager (LCM); cleanup of transferred logs) on the SD.

As part of the log tran:~fer process, the LCM begins a secure log transfer to the SM with the date, device type, and SD
name for the log being i~ransferred.
SDs having been assigned hostnames/aliases that indicate their security function and geographical location are then categorized into SD lisps associated with the LCM(s) in the system.
When an LCM contacts th~~ DAM, the LCM is provided with the log retrieval, and log ~~leanup configurations, as well as 1~) the SD list for which that LCM is responsible.
As the LCM(s) notify the DAM of the successful transfer of SD logs, the DAM then contacts the SM for the location of the SD log such that t;he appropriate data filter can be applied to the log.

Storage Manager This section contains the detailed design information for the Storage Manager (SM), which is a component of the Security Devices Log Reporting System (SDLRS). The SM is responsible for the management of physical log archival storage/access, and the corresponding interaction with the Data Analysis Manager (DAM) and Log Collector Manager (LCM) components of the SDLRS.
Design Representation The intent of this section is to provide the architecture and design of the SM and not the implementation specifics of the SM. For ease of understanding, the SM system configuration detailed in the design is provided to highlight key fields and information that are required by 1~ the SM. The actual implementation of the content may vary.
Overview Major Functions of the Storage manager Receives Security Device (SD) logs from the Log Collector Manager (LCM) for system archival.
Management of online and offline log archivals, and the transition of logs frc:~m. online to offline status.
Provides the Data Analysis Manager (DAM) with access to SD
logs upon request.
Provides the DAM with access to the SM log archival tables upon request.
CORBA Integration The Storage Manager (SM) acts as both a CORBA client and a CORBA server. The CORBA interface for' the SM is defined in the SDLRS CORBA integra~t.ion document The service requests ?0 that are defined in the CORBA integration document are referred to in this document whenever possible. They will appear as the actual interface method name preceeded by "SM-". For example, the service request represented by the SM
providing logfile information to the DAM is SM-GetLogInfo.
Service Request Function Prototype The service request functions are based on the service requests defined in the SM entity interface in the SDLRS
CORBA integration document [MA1].
System Variables For UNIX-based SM implementations, the system variables are analogous to the UNIX shell environment variables (e. g.
1o setenv in the csh) and can therefore be used for that purpose (e. g. setenv SMDIR <DirLoc>, for the csh).
ARCHIVEDIR . Dirloc The ARCHIVEDIR variable defines the location of the directory used to archive online logs according to their 15 security device type.
RESTOREDIR .- DirLoc The RESTOREDIR variable defines the location of the SM
restored logfile directory. This is the location where offline logs are to be restored to disk.
2o SMDIR . Di.rLoc The SMDIR variable defines the location of the SM run-time directory, which contains the: configuration files; online and offline archival tables; log reference tables; restored log archival table; potentially other configuration files.
25 This variable symbol i~; also used as a production in syntax definitions in this document.
SMDIRBKP . DirLoc The SMDIRBKP variable defines the location of the SM
configuration backup directory located on a different disk partition than that of the SMDIR directory. The primary reason for SMDIRBKP is to maintain a second copy of the log archival tables which are of a highly dynamic nature.

Configuration Repository The SM configuration repository at version 1.0 will be a configuration file. It is .Located on the SM host and uses the following syntax:
SMConfigRep . SMDIR "/" "SM.ini"
In future versions of SDLRS, the SM configuration repository may also be available v:ia a database table. If the SM
configuration repository is a database table, then it will use the following synta:K:
1o SMConfigRep . "SMConfig"
Initialization The SM queries the DAM for the log archival interval configurations for the different device types.
The SM validates that t:he appropriate online and offline 1~ archival tables exist based on actual device_types (i.e.
EntityTypes) for currently archived logfiles.
The SM checks the pending activity file to see if it has any pending actions to execute or restart.
The SM performs any necessary log cycling from on-line 2o status to off-line status, and from off-line status to N/A
status.
Log Archival Tables A logfile has an archival status of either "online" or "offline". This archival status must be maintained along 25 with other logfile attributes for as long as the logfile exists within the system. To do this, an archival table is maintained for each type of security device's logs that are managed by the SM. The two exceptions to this are: 1) export controlled devices; 2) logfiles that have been 3o previously offlined, anal then restored. In each of these cases, separate tables are maintained, however, the table and record format in each case is identical. Maintaining a separate archival table for each security device type, allows for greater scalability of the aystem, which in turn will enhance the performance of table and logfile retrievals on a large s~~stem with many different types of security devices.
Archival Table Naming Convention The security devices archive table name syntax is as follows for non-export-controlled security devices:
EntityTypeArchTbl . EntityType Hyphen "ArchTbl"
1~ EntityType . as defined under "Modules" -CORBA integration [MA1]
The archive table name syntax for export-controlled security devices is as follows:
ExpEntityTypeArchTbl . "Exp" hyphen EntityType hypen"ArchTbl"
EntityType . as defined under "Modules"
CORBA integration [MA1]
The archival tab:Le name syntax for restored "offline"
security device :Logfil.es is as follows:
2o ResEntityTypeArchTbl . "Res" hyphen EntityType hyphen "ArchTbl"
EntityType . as defined under "Modules" -CORBA integration [MAl]
Creation of New Archival Tables The security devices for which archival tables exist are defined within the system by the 'Enti.tyType' , as defined under the "Modules" section in the CORBA integration document [MA1]. The ~~M will create a new security device archival table if one cLoes not already exist under the following conditions:

Upon receiving a logfilf=_ from an LCM, the security device type is extracted from the security device hostname alias, and validated against known "Entity Types". If this is the first instance of a valid security device type log archival, then an archive table i;s created for the security device type.
An event which leads to the creation of a security device archival table will result in an alarm being generated and sent to the DAM via DAM-Event as documented in the CORBA
1) integration document [MA1].
Archival Table Format Determining Security Device Type The security device type associated with a table is determined by parsing the archive table name. For example, 15 the firewall archive table name would be "FW-ArchTbl". The export controlled firewall archive table name would be "Exp-FW-ArchTbl".
Archive Table Characteristics An archive table is a chronologically ordered table based on 2o the date and time of the actual log archival occurring on the SM. For this reason, no inserts to the table are required, as all new records will be appended to the table.
The table is of fixed record size.
The table contains records for logfiles with online status 25 as well as offline status The first two records of an archive table are reserved for table specifics. These specifics should include as a minimum:
Table offset for the first "Offline" archival record, and 3o the "logfile date" associated with the archival record Table offset for the first "Online" archival record, and the "logfile date" associated with the archival record The records between the first archival record with an "Offline" status and the first archival record with an "Online" status, are logfiles deemed Offline.
The records following the first archival record with an "Online" status are logfiles deemed "Online".
One archive record exists per archival directory regardless of the number of logfi.les contained in that archival directory. The number of logfiles expected within an archival directory to be determined by the logfile retrieval-per-day interval.
Archive Record Format A log archival record consists of the following required fields:
Directory Reference-ID // unique path of the directory 15 containing a logfile(s) Logfile Date // date of logfile created by security device Online Status // either Online, Offline, or Restored Logfile Type // correlates to the type used by the 2o data filter Retrievals_Per_Day // correlates to the # of logfiles per unique directory SD Name // security device alias name Data required for transaction audit purposes. This data 25 would be relatively static for a device and hence may be better accessed through the SDLR logging. However, they are included here as optional fields within an archival record:
SD IP Address // security device's IP address LC Name // the name of the Log Collector 30 LCM Name // the name of the Log Collector Manager Archive Record Example The following is an example of a log archival record for a non-export-controlled f:irewall including the required and optional fields in the :record:
Directory_Reference_ID . "unique hash of DirPath" where Dirpath =
$ARCHIVEDIR/Main/Wk Of-the Month/Device_Type/Logfile Date/SD
Name Logfile_Date - 19991210 1~ Online Status - "Enum type for Online"
Logfile Type - EAGLE
Retrievals_Per Day - 1 SD Name - fw-1-n-cn SD IP Address - 47.1.2.3 15 LC Name - <hostname>
LCM Name - <hostname>
Logfile References A "Logfile Reference" is used to uniquely identify a logfile 2o archived on the SM. "Logfile References" are utilized by the SM for tracking logfiles requested by the DAM in either automated analysis mode or custom analysis mode.
Each "Logfile Reference" consists of a "Directory Reference ID" component and a "Logfile Name" component. Taken 25 together these components comprise a Logfile Reference ID, which uniquely identifies a logfile archived on the SM.
Logfile Reference ID
Logfile Reference IDs a.re used by the SM in its communication (Open Method) between the SM and LCM objects, 30 between the LCM and DANf objects (interface DAM-LogArchDone), and in its communication (Open Method and the interface SM-GetLogInfo) between the SM and DAM objects. The two parts which make up a "Logfile Reference ID" are the "Directory Reference ID" and the ":Logfile Name" .
Directory Reference ID
The Directory Reference ID is a unique "hash number" based on the archival direct.o:ry where a logfile will reside.
Depending on the hashing algorithm used, the unique "hash number" may be of varying lengths. However, the 'hash number' should not exceed 128 bits so that it does not negatively impact the size of archival table record entries 1o where the Directory Reference ID is stored.
The archival directory to be hashed is of the following format:
E.g., Export-controlled Security Device directory $ARCHIVEDIR/ExpCtl/Wk-Of The Mon/Device Type/Logfile Date/SD
15 Name E.g., Non-export-controlled Security Device directory $ARCHIVEDIR/Main/Wk Of:_The Mon/Device_Type/Logfile Date/SD N
ame Logfile Name 2o The "Logfile Name" component of a "Logfile Reference" is comprised of the "Logfile Type" associated with a logfile, and a sequencing number. The boundaries of potential sequencing numbers determined by the "Retrievals-Per Day", and the existence of logfiles with sequencing numbers 25 already contained within the archival directory.
The "Logfile Name" is of th.e following format:
Logfile Type hyphen <:~equence number> hyphen "log" period <compression tag>
E.g Firewall logfile of type EAGLE and a retrieval interval of 2 provides up to two possible "Logfile Names". With the GNU compression tag being used in this example, the two potential "Logfile Narnes" are:

EAGLE-1-log.gz EAGLE-2-log.gz Logfile Reference ID Format A logfile is uniquely identified by combining the "Directory Reference ID" with a "Logfile Name". An example is given below:
E.g. Logfile Reference ID for a unique firewall logfile of type EAGLE and a Retrieval Interval of 1 <16 bit hash of archival directory> .EAGLE-1-log E.g. Logfile Reference ID for all logfiles of a firewall of logfile type EAGLE and a Retrieval Interval of 4 <16 bit hash of archival directory>
Directory Reference IL) Index A "Directory Reference ID" index is maintained for each archival table. As each "Directory Reference ID"
identifies a unique archival record, the index is used to facilitate archival record lookups by associating the unique "Directory Reference ID" to the offset in the table where the archival record is stored.
Log Archival Disk Archival New logs for archival acre received from a LCM via the Netfile methods (i.e. Open, Put, Close) defined in the SDLRS
CORBA Object Architecture document [MA1]. The process of archiving a log is deta~i_led below:
The SM checks for enough available disk space to receive the log in its entirety.
.o Based on the security device type, logfile date, and device alias, the appropriate archival directory is created if required and the logf.il_e is received. (See the Logfile References section for the archival directory and logfile name formats.) Upon receipt of the logfile(s) for the security device, a new entry is created in the applicable security device Archival Table if this is the first logfile to be stored in .
the archival directory.
Tape Archival With online data archiving, the potential exists for large volumes of data to reside on the SM archival disks. This 1o data can be broken down into dynamic data (e. g. newly archived logs) and static data (e. g. previously archived logs). To reduce the cost associated with tape archivals, it is therefore useful to architect the log archival directories/disks in such a manner that full backups of 15 static data occur only once, which is at the time that the data volume becomes static. Incremental backups are then done on a nightly basis to backup any new logs archived that day.
Facilitating Low Cost Tape Archivals by Archival Directory 2o The SM will have incremental tape backups on a nightly basis and full backups of static archival directories/disks on a weekly basis. To facilitate this functionality, the online log archival filepaths will reflect the week of the month that the logfiles were generated. The weeks are defined as 25 follows:
wk1 . days 1--7 wk2 . days 8--14 wk3 . days 15-21 wk4 . days 22-28 + days 29,30,31 as required 3o Some example log archival filepaths are as follows:
1) Logfile_Date = 19990804; Security Device Type = fw Logfile_Path = $ARCHIVEDIR/wk1/fw/19990804/fw-1-n-cn/EAGLE-1-log.gz Logfile Date = 19990812; Security Device Type = fw Logfile_Path = $ARCHIVEDIR/wk2/fw/19990812/fw-1-n-cn/EAGLE-1-log.gz 3) Logfile_Date = 19990829; Security Device Type = fw Logfile_Path = $ARCHIVEDIR/wk4/fw/19990829/fw-1-n-cn/EAGLE-1-log.gz A weekly full backup tape archival can. then be setup to archive $ARCHIVEDIR/wk[n] ( where n=[1~2I3~4]) on a rotating basis. The rotation is based on the full backup to be done of the archival directory for the proceeding week. The effect of this rotation is to reduce t:he incidence of reoccurring full backup tape archivals of static data.
An example of a weekly full backup scenario is given below:
Sunday, August 31 full backup scheduled 2o day of backup = 31; days/wk = 7 31 div 7 = 4 proceeding week = (4 -1) - 3 ; (In the case where the proceeding week is less: than or equal to 0, the proceeding week becomes equal to 4:. ) 2,5 full backup of $ARCHIVE;DIR/wk3 to tape Accessing Logs and Log Archival Tables Logfiles once they are stored on the SM can be accessed via the DAM interface to the SM either as part of the automated analysis mode, or the 7_ogfi.les can be accessed by the WAS
via the DAM interface t:o the SM as part of the custom analysis mode. Logfile Archival Tables can also be accessed by the WAS via the DAM interface to the SM.

Accessing Logs Automated Analysis Mode In the automated analysis mode, a Directory Reference ID
(DRID) and a log Archival Table entry are created at the 5 time that the LCM successfully completes its transfer of a SD logfile(s) to the SM. The Logfile Reference ID (LRID) are passed back to the LCM, so that they may be passed to the DAM as part of the LCM log availability notification process (i.e. DAM-LogArchDone). The DAM then will provide 1o the LRID as part of a log retrieval request to the SM.
Custom Analysis Mode Obtaining Logfile Information In the custom analysia mode, a request is received from the DAM (i.e. SM-GetLogInfo) in which logfile information is 15 passed in the request.. The SM then returns the requested logfile records from th.e associated security device Archival Tables.
Logfile Retrieval An actual logfile retrieval request in custom analysis mode, 2o will provide a Logfile Reference ID (L~RID), which can uniquely identify a log~file for retrieval or a set of logfiles applicable to a security device for a particular date. The LRID for a unique logfile contains both a DRID
and Logfile Name component. The LRID for the logfiles of a 25 specific date may contain only a DRID component.
As it is possible in the custom analysis mode to have several concurrent requests for a particular logfile at one time, the SM must manage each log transaction independently from another.
'o Log Archival Tables The retrieval of archival tables is based on three factors:
security device type; whether the devices are export-controlled or not; whether the archival tables are for restored logfiles or not. A request from the DAM to return archival table entries is made through the SM interface SM-GetLogInfo as defined in the CORBA integration document LMA11.
Transitioning the Archival Status of Logs Logfiles are archived on the SM for a specified online archival duration (e. g. by default the duration is three to months). After the online archival period, a logfile record is tracked for the duration of the offline archival period.
The time of the offline archival period being dependent on whether or not the security device is an export-controlled device. After the offline archival time period has transpired, the recox-d of a logfile is no longer tracked.
Transitioning Occurrence The transistioning of archival status from offline to N/A
is done on a nightly basis, as it is essentially an archival table manipulation operation only. The transistioning of 2o archival status from online to offline is done on a weekly basis rather than a daily basis. The advantage of this weekly processing is th.e ability to have archival transition occur on a day where log data volume is expected to be lower (i.e. Sunday) than during the rest of the week. The disadvantage to weekly processing is that approximately 86~
of the logs will be archived online far an average 3 days longer than the configured monthly archival rate, which will result in a slight increase in the disk space required for online archival. For example, using the default three month online archival rate (90 days), an extra three days would necessitate an approximate 4o increase in disk space requirements.

Offline to N/A Status Once a day, the SM transitions logfile records from offline status to N/A status. The SM does this in the following manner:
The first record of an archive table contains the offset of the first offline archival record.
Beginning with the first offline archival record, the SM
sequentially examines the "logfile date" of each archival record to see if it meets the offline archival duration criteria.
The offset of the first archival record which meets the offline archival duration criteria is saved along with the "logfile date" to the first record of the archival table.
Online to Offline Status Once a week, the SM transistions any logfile archival records that exceed the online archival duration to offline status. In the process, the SM also compresses the archival table by removing archival records that have exceeded the offline archival duration period, as well as rebuilds the Directory Reference ID index. The SM does this in the following manner:
The first record of an archival table contains the offset of the first offline archival record. The second record of an archival table contains the offset of the first online archival record.
The Directory References ID index associated with the archival table is recreated at the time of creating the newly compressed archival table.
Beginning with the fir~:t. offline record, the archival table 'o is rewritten. Logfile archival records are written to a temporary table with an updated offline status up to the first online archival record. (The offset of the first online archival record is provided in the second record of the archive table.) Each subsequent archival record is then examined as to whether the 'logfile date' has exceeded the online archival duration period. If ~.t has, and the ' 'logfile date' directory for that security device type (i.e. EntityType) exists, the 'logfile date' directory is removed. The archival record is then. written to the temporary table with an offline status.
1o The offset of the first archival record whose "logfile date"
meets the online archival duration criteria is saved to the second record of the archival table, and the record written to the temporary table with an online status.
All subsequent record~> are written to the temporary table as is with no archival duration comparisons required.
The temporary table replaces the current table.
Restored Offline Logs Restoring a Log 2o Restored logfiles from backups are differentiated from logfiles that are still. online in order to make it easier to track them from an administrative perspective. Therefore, log restores will be restored to the RESTOREDIR/Newlogs directory.
~5 Processing Log Restore;
The SM on a hourly basis checks the RESTOREDIR/Newlogs directory for newly restored logfile directories. For each restored logfile directory, an archival record is created in the applicable "restored" archival table, (see Log Archival Tables section for more details) and the logfile directory is moved to the appropriate RESTOREDIR directory. An example is provided below:

Regular archival path:
$ARCHIVEDIR/Main/Wk Of The Mon/Device_Type/Logfile Date/SD N
ame Restored archival path:
$RESTOREDIR/Main/Wk Of The Mon/Device_Type/Logfile Date/SD_N
ame A notification indicating that the logfile has been restored is then sent to the DAM: via DAM-Event as documented in the CORBA integration document [MA1].
Transitioning Restored Logs Restored logfiles are kept online for the restored logfile duration period, which has a default duration of one month.
Restored logfiles are transistioned directly from online to N/A status on a weekly basis in the following manner:
As all archival records of a restored log archival table deal with online logs, the first two records used to store the first offline record offset, and the first online record offset are not required, to be utilized..
The same process of recreating the associated Directory 2o Reference ID index and archive table a.s that for non-restored log archival tables is used but with the following difference. For each restored online. archival record, the logfile's "last access>" timestamp is used to determine if the restored logfile duration has been exceeded.
Concurrent Event Handling The nature of the SM is; that it will not have to deal with a large number of transactions-per-second (tps), but rather that the majority of SI~I transactions will be of a long-lasting nature due to event-caused, prolonged, disk-related activity. Given these system specifics, the SM must be able to handle multiple concurrent events. For example, a "transfer log to SM" notification from an LCM (SM SR-2), and a "transfer log from SM" notification from the DAM (SM SR-3) can arrive at the same time. Each of these events could potentially result in substantial disk activity given that logfiles can be of substantial size.
The most efficient means of handling concurrency in this scenario is through lightweight threads. In the worst case of the SM running on a single processor system, the overhead involved in thread creation and in context switching between 1~) threads is minimal when compared to the latency times associated with disk accesses. In the best case, of multiple disk controllers, and multiple processors on a SMP
(symmetrical multi-processing) SM system, threads would be able to concurrently process on different processors/disk 15 controllers. For these reasons, the SM should be implemented using threads rather than :by an event loop.
Activity Status File 2~7 The Activity Status File (ASF) contains state information for various activities going on in the SM. For example, as each logfile transfer notification from the LCM is received, the SM stores the event related information so that if the system crashes, it can restart any pending activity.
25 The information in the stat file can be displayed via SM-GetStatus.
The pending activity file syntax is:
StatFile . SMDIR"/" "SM.stt"
The syntax for each record is:
3~7 ASFEntry . JoblVumber Activity JobNumber . Into=ger[4]
Activity . Archival I Access Archival . Status ";" DateTime ";" SDName ";"
LCMName ";" \
LogRefs Access . Status ";" DateTime ";" SDType ";" SDName ";"
SearchAttr ";" LogRefs ";"
Status . "n." // new but not acted on """ // started job "c" // complete, just cleaning up to ~ ":E" // failed, just cleaning up ":r" / / system failure, j ob restarted NOTE: The "r" status applies to the automated analysis mode 15 since the custom analysis mode is predicated on an initial web browser request to the DAM.
Storage Manager Event Logging The SM logging uses sys:Log. Syslog should be setup with the following parameters:
20 void openlog(const char *ident = "SM", int logopt =
LOG-PID+LOG_NOWAIT, int facility = LOG USER;I;
A message will be issued when the following occurs:
2~~ SM starts up, including command line parameters SM shuts down SM receives SM-GetLogIn:Eo including all parameters SM receives SM-GetStatu:~ including all parameters SM receives SM-SetConfigInfo including all parameters 30 SM receives SM-Noop SM receives NetFile method r_alls (Open, Get, Put, Close) and associated parameters Significant state changes during archival jobs (e. g. start, end, misc.) Significant state changes during the creation/transitioning of archival tables Security related events When an error occurs Initially the message part of the syslog() call should be in a machine parsable form. In the future, the message format should follow the Nortel Networks Common Logging Format.
1o For Future Study A tool to enter and modify SM configuration information.
Archival transitioning ~ger individual security device type archival duration configuration.
Use of a Common Logging Format 1 !i Appendix A: SM Design Information From SDLRS Design Document The following is the SM des:ign information from the SDLRS
specification design description above.
Storage Manager (SM) 2u The SM is responsible for SD log archival in the correct location, maintaining an index of log archivals according to SD and export control configuration settings, and backups of the log archiving system. As part of the log transfer process, the LCM begins a secure log transfer to the SM with 2~~ the date, device type, and SD name for the log being transferred. From this information, the SM then selects the appropriate on-line archival directory where the log will be written. Upon successful completion of the log transfer, the SM then updates its indE=_x of log archivals.
3U To manage the transition of logs from on-line to off-line archival, the SM receive=_s from the DAM the log retention configurations for the ;system on a daily basis. By default the log archival configurations are set at the following:
perimeter devices - 3 months on-line and 15 months off-line;
export controlled devices - 3 months on-line and 57 months off-line; drop-box devices - 3 months on-line and 15 months off-line; devices classified as "other" (e.g. SPAM logs) - 3 months on-line. The SM i~hen manages the transition of on-line log archival to of:E-line archival by performing disk cycling, off-line archival backups, and the updating of the log archival index.
Upon receiving log location requests from the DAM, the SM
references the archival index for the location of the log.
If the log is on-line, i~hen the file path is given to the DAM. If the log is found to be off-line, then the DAM is informed that the log i:~ off-line. Archival information for specific SD logs or for the complete on-line or off-line indices can be provided to the DAM on request.
The DAM is responsible :Eor providing the configuration details to the other sy:~tem components, ensuring that all SD
logs are archived, performing data analysis on SD logs, 2o providing summary stati:~tics to the Data Analysis Store (DAS), and querying the SM for log archival information upon request.
Logs that have been sec,.~rely pulled, are then securely pushed to the Storage M<~nager (SM) for archival with the LCM
2~~ providing for each log l.ransfer the device type, date, and SD name to the SM.
As the LCM(s) notify the=_ DAM of the successful transfer of SD logs, the DAM then contacts the SM for the location of the SD log such that the appropriate data filter can be 3o applied to the log.

Horne 1380680 Glossary DAM: Data Analysi:~ Manager - the system component which synchronizes the overall system, and performs the analysis on logs.
DAS: Data Analysi;~ Store - the system database where the system configuration .and summary metrics are stored.
IMMUNEsystem: Intrusion Monitoring and Management of Unified NEtworks system - an enterprise security environment of which the Security Devices Log and Reporting System is a to part .
LC: Log Collector - the system component which directly interfaces with a Security Device logging mechanism.
LCM: Log Collection Manager - system component which manages the collection of all Security Device logs and transfers the logs t.o the the Log Archival Unit.
SD: Security Devices - devices used by the enterprise to manage data security within the enterprise network.
SDLRS: Security Dev_Lr_es Log and Reporting System SM: Storage Manager - system component responsible for log archival, ad-hoc :Log retrieval, and backups.
WAS: Web Application Server - contains the applications which provide the system and data interfaces to the user.
WS: Web Server - the user's access point into the system WC: Web Client -- a web browser capable of interfacing with the web server for data presentation to the user.
SECOPS:
SECINV:
SPAM:
DRID Directory Rf=_ference ID

LRID Logfile Reference ID
LEL Log Exception List LTL Log Transfer List SDF Security Device File LCT Log Collector Table SDT Security Device Table RAS Remote Access Services Horne 1380680

Claims (22)

WHAT IS CLAIMED IS

1. A security device log and reporting system for a data network, comprising:

a Log Collection unit, for collecting log files from security devices, a Data Analysis and Log Archival unit for analysis and archival of log files, and a Data and System Access Unit providing a user interface with the Data Analysis and Log Archival Unit.

2. A system according to claim 1 wherein the Log Collection unit comprises a Log Manager for managing log collection from a plurality of security devices.

3. A system according to claim 1 wherein the Log Collection unit comprises a plurality of log collectors and a log collection manager for managing log collection from a plurality of log collectors.

4. A data network security management system for security device log archival and reporting comprising:

a log collection units comprising a plurality of log collectors, each for collecting log files from a plurality of security device node and a log manager for collecting log files from a plurality of log collectors;

a data analysis and log archival unit for archival and automated analysis of log files received from the log manager.

and a data and system access unit providing a user interface to the Data Analysis and Log Archival Unit.

5. A system according to claim 4 wherein the log collector provides output to a storage manager and a Data Analysis manager, connected to a Data Analysis Store, of the Data Analysis and Log Archival unit, which also comprises a Archival unit associated with the Storage unit.

6 A system according to claim 4 wherein the user interface is a web based user interface

7. A system according to claim 1 wherein the data and system access unit wherein the user interface provides for log analysis summaries, trend analysis, controlled operational access and system configuration

8. A system according to claim 1 wherein the access unit comprises an authenticated, authorized, secured web based system.

9. A system according to claim 4 wherein the log collector receives logfiles from security devices comprising one or more device types including:
Firewalls, (raptor 4, Raptor 6, CES Checkpoint Firewall-1) Remote access service; (RAS), CES (Contivity Extranet Switch), SPAM (Mail shield), FTP Drop Box and Anti-Virus (Antigen)

10. A system according to claim 1 wherein the Log Manager LM
interfaces with a Data Analysis Manager (DAM) and a Storage Manager (SM) and the LM comprises:
means for collecting logfiles from security devices, means for pushing cached SD logfiles to a Storage manager for archival, and means for providing log archival status updates to a Data Analysis Manager (DAM).

11. A Log Manager for a data network security management system, wherein the Log Manager LM interfaces with a Data Analysis Manager (DAM) and a Storage Manager (SM) and the LM
comprises:
means for collecting logfiles from security devices, means for pushing cached SD logfiles to a Storage manager for archival, and means for providing log archival status updates to a Data Analysis Manager (DAM).

12 A system according to claim 1 wherein the Log Collector Manager (LCM) interfaces with a Data Analysis Manager (DAM) and a Storage Manager )SM) and the LCM comprises:
means for receiving logfiles from the plurality of log collectors, means for obtaining a logging system configuration from the DAM, means for propagating the configuration to individual LC
associated with Security devices, means for providing notification to the LC to begin transfer of SD log files, and means for pushing cached SD log files to the Storage manager for archival, and means for providing log archival status updates to the DAM.

13. A system according to claim 1 wherein the Data Analysis and Log Archival unit comprises a Storage Manager (SM) and a Data Analysis Manager (DAM) and the SM comprises means to receive security device logs from the Log Collector Manager, means for system archival, means for management of online and offline log archivals and transition of logs form online to offline status, means to provide the Data Analysis Manager (DAM) with access to SD logs on request, and means to provide the DAM with access to the SM log Archival tables on request.

14. A security device log and reporting system wherein archival of log files is separated from analysis of logfiles.

15. A security devices log and reporting system comprising a Log Manager, the Log Manager having a distributed interface for receiving logfiles from a plurality of security devices, and is the interface to a Data Analysis and Archival unit of the system.

16. A security device log and reporting system according to claim 15 wherein the Log manager comprises an intermediary caching system for log files received from the plurality of security devices.

17. A security device log and reporting system according to claim 14 comprising an Data Analysis and Archival Unit, a Log Collection Unit comprising a Log Manager, and Data and system Access Unit, wherein Data Analysis and Archival Unit interfaces with only a Log Manager and a Data and System Access Unit, whereby interfaces are easily protected via a firewall and instrusion detection system.

18. A method of managing security device log archival and reporting for a data network security , comprising collecting log files from a security device node at a log collector collecting log files from a plurality of log collectors at a log collection manager transferring log files from the log collection manager to a data analysis and log archival unit for archival and analysis.

19. A method of managing security device log archival and reporting for a data network security , comprising collecting log files from a security device node at a log collector collecting log files from a plurality of log collectors at a log collection manager transferring log files from the log collection manager to a data analysis and log archival unit for archival and analysis, logfile analysis being separated from log file archival.

20 A method according to 18 comprising providing user access to the Data analysis and log archival unit via a a data and system access unit.

21. A Storage Manager for a security device log archival and reporting system comprising means for receiving security device logs from the log collector manager for system archival, means for management of online and offline log archival and transition of logs from online to offline status, means for providing the DAM with access to security device logs on request, means for providing the DAM with access to the SM log archival tables on request.

22. A storage manager according to claim comprising means for differentiating types of log files.