WO2007066166A1

WO2007066166A1 - Method and system for processing and displaying maintenance or control instructions

Info

Publication number: WO2007066166A1
Application number: PCT/IB2005/003709
Authority: WO
Inventors: Charlotte Skourup; John Pretlove; Thomas Pettersen; Christoffer Apneseth; Pierre ÖBERG
Original assignee: Abb Research Ltd
Priority date: 2005-12-08
Filing date: 2005-12-08
Publication date: 2007-06-14

Abstract

A method to process and display control instructions and technical information for an equipment, plant or process in an industrial facility. A software entity may be configured with identities of selected said equipment, plant or processes. The software entity may also retrieve information associated with said equipment, plant or process by means of being so configured. Information may be combined and annotated on a display device to provide control or maintenance instructions. A display device, a computer program and a control system are also described.

Description

Method and system for processing and displaying maintenance or control instructions

TECHNICAL AREA

This invention relates to an improved method and system for providing, processing and displaying maintenance and/or control instructions for an equipment, plant or a process by means of accessing data and stored information and displaying information relevant to the device or process. The invention provides improved means to generate and display instructions necessary to configure, maintain or repair equipment.

TECHNICAL BACKGROUND

Monitoring and control of plant and equipment in industrial facilities is largely automated and computerised. A traditional ^"automation system is one in which each physical asset is described in a database. In its turn, this database is

frequently modeled as a multitude of tables, and relationships between different fields and tables in the database. Although much technical information about various devices/ equipment, process sections etc exists, it is most often distributed among different systems each having different categories of data and/or different methods of storing and retrieving technical data. Examples of such existing and often unconnected systems are computerized maintenance systems, process control systems, power management systems, energy management systems, and systems for process simulations and optimization.

Examples of information to be found in such systems are

maintenance histories, production histories, configuration set- points, production flows, compressor efficiency curves,

references to load characteristics of drive solution, energy consumption logs, ratings for electric motors, information about material costs and energy costs. An installation may also include equipment from different suppliers and from different industries. It is a complex and difficult task to retrieve information from all of those systems so as to give a reliable and extensive picture even on a historical basis. It is also very difficult, time consuming and error prone to organize, enter, maintain and retrieve information related to a specific device. It is even more difficult to retrieve and or access such information when an alarm or other event is reported.

From the international patent application WO 01/02953 entitled Method of integrating an application in a computerized system, it is known to represent real world objects in control systems by means of one or more software components. The real world object may be a single device, an object in a process or complete equipment.

Use of augmented reality in industrial control or monitoring is known from WO 2005/066744 entitled A Virtual Control Panel, in which it is described how a computer produced image, an

augmented reality image, may be overlaid on a real world image. Manipulation of the composite image comprising real

world/augmented reality data may be carried out to provide an interaction with a control device or system.

However the prior art does not provide comprehensive access to plant, device or process section information in a timely and simple way that supports fast or on-line operations. In

addition, technical documentation and support materials for maintenance operations, repairs, configuration and/or other technical adjustments to a process, plant, or other production or processing site are often stored in different print and or digital media and are also often difficult to keep up-to-date. SUMMARY OF THE INVENTION

According to one aspect of the present invention, one or more embodiments of the invention provide an improved method for processing and displaying control or maintenance instructions for an apparatus, device or equipment in an industrial process or facility comprising one or more control systems for

monitoring and control, wherein device or process-related information and other data for each said equipment, plant or process are stored in and may be retrieved by said control system, the method comprising identifying a said equipment, making and processing an image of a first view of the said equipment, making, retrieving or recording an annotation of control or maintenance instructions for said equipment and arranging the annotation represented as a graphic element for a display, and by combining the first view of the image and the annotation into a composite image, and displaying the composite image of said equipment combined with additional control or maintenance instructions to a user.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, device or equipment in an industrial process or facility comprising one or more control systems for monitoring and control, wherein device or process-related information and other data for each said equipment, plant or process are stored in and may be retrieved by said control system, the method comprising recording an annotation and wherein the annotation for said equipment is placed in a selected position relative to the first view of the image .

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, device or equipment in an industrial process or facility comprising one or more control systems for monitoring and control, wherein device or process-related information and other data for each said equipment, plant or process are stored in and may be retrieved by said control system, the method comprising generating the annotation as a part of a layer for superposition on the first view of the image.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, device or equipment in an industrial process or facility comprising one or more control systems for monitoring and control, wherein device or process-related information and other data for each said equipment, plant or process are stored in and may be retrieved by said control system, the method comprising arranging the annotation displayed relative a position on first view of the image or relative a position on the display screen.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising displaying an instance of the composite image

displayed on a second display to a second user or expert.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising recording an annotation which provided by an input action of a second user or expert such as manipulating a second instance of the composite image displayed on the second display. According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising adding one or more annotations are added to the composite image and displaying the image to a plurality of other users .

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising in part providing control or maintenance instructions used to configure a set point or switch a device on or off by means of software means of part of the composite image.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising in part activating a software means and thereby providing control or maintenance instructions used to generate one or more control signals.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising identifying said equipment by image processing a unique marker means on or near said equipment .

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising identifying said equipment by image processing images provided by the optical means or by processing the images in part by scanning natural features of the images. According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising changing the position and/or orientation of the annotation in the composite image relative a distance or size relationship in a second view of the image dependent on a distance or size relationship determined with an image of a first view.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising changing both the position and size of the annotation relative to, and depending on, a second view of the image dependent on the size relationship determined with the first view.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising identifying said equipment by processing

transmissions from one or more wireless node means arranged on or near said equipment.

According to another embodiment of the present invention an improved method is provided for processing and displaying control or maintenance instructions for an apparatus, the method comprising generating instructions and information for carrying out a repair, re-configure, re-programming or replacement of a faulty part of said equipment, plant or process.

According to another aspect of the present invention, one or more embodiments of the invention provide an improved system for processing and displaying control or maintenance instructions for an apparatus, device or equipment in an industrial process or facility comprising one or more control systems for

monitoring and control, and one or more computers, wherein device or process-related information and other data for each said equipment, plant or process are stored and may be retrieved from said control system, and comprising an AR system comprising a tracking means, a display means, at least one computer means, a data communication means for communicating an image of a first view of said equipment, and computer software image processing means for identifying the said equipment in the industrial process or facility, and computer software means to record control or maintenance instructions comprising any of text, video or voice messages and computer software for attaching said message information, and computer software means to provide a composite image of an image of said equipment combined with additional control or maintenance instructions to a user.

The invention comprises an improved system for processing and displaying an augmented reality representation of computer- generated graphical information, generated and stored in advance, overlaid the real world. Such an improved system may also comprise a communication system supporting voice

communication and transfer of the entire, or parts of, augmented reality interface and/or additional virtual information.

In summary, such an improved augmented reality system may comprise: a handheld interacting and pointing device with a tracking system for determining its position and orientation in relation to a world coordinate system and a portable display device eg preferably a wearable device such as glasses, head- mounted display or head-up display or else a PDA, notebook, tablet computer, or similar, for visualizing augmented reality overlaid the view of the real world. In the event that video see-through is used for augmented reality visualization, the display device further comprises: a camera, such as a video camera of some sort for capturing a stream of images of the environment mounted on or integrated with the display device. The camera is mounted in a fixed position at the display device, and the display device is preferably located along the camera view axis and at the camera's image plane. The system combines computer-generated graphics and annotations with the live video stream and projects the combined augmented reality video onto the display device. The user sees the video with overlaid virtual information as if he was looking at the real world. For optical see-through AR, the computer-generated graphics are registered directly onto the display device and follow, on the image viewed, and on the user's view of the real world. The virtual graphics are overlaid on the optical image of the real world without including a video the real world. One or a series of still images may be used to carry out the same or similar steps or actions with the stream of video images described above .

To identify the equipment of interest, unique identity markers, ID markers, may be placed at objects (equipment, devices and systems) in the environment. These are preferably processed by a recognition system for recognizing the unique IDs and use those IDs to identify a known device or equipment controlled by a control system. Further a tracking system is included for determining position and orientation of a tracking device in relation to a world coordinate system. As well, or as an

alternative, a recognition system for recognizing and tracking one or more visible objects at a physical location may be used. For example a vision-based tracking system using a recognition or tracking technique known as Natural Feature tracking may be used. For more detail about a type of natural feature tracking see for example an article entitled: Natural Feature Tracking for Extendible Robust Augmented Realities, Park, You & Neuman, UCLA, in a paper for the first International Workshop on

Augmented Reality (IWAR) June 1998. This type of technique may be used with image processing to recognise one or more objects in the real world and supply information about viewer-to-object distance, and changes in that distance, so that a position of the user in relation to certain real world objects may be determined. The determinations or calculations may then be used provide a graphic display depending in part on how the position of the user and the orientation of the user changes relative to a given real object.

The processing of the image may be speeded up by combining a calculation of a movement path of the camera/viewer based on changes in the viewer-to-object distance for one or more objects. Calculations are advantageously carried out by

recognising in the visual images natural features such as corners, edges, or other higher-contrast features identified in a first view and then, in successive views, calculating a movement path and orientation based on the viewer-to-object distance for features already identified in the first view or other earlier scan. In this way, by identifying initially and then not carrying out the identifying processes each time, or at least not in a foreground process, and calculating from the change position in the image if the natural feature (s) already identified a movement path and orientation of the user which enables quick identification of an equipment in a surroundings and faster processing, faster refresh for example, of the composite image.

A major advantage of the present invention is that a maintenance action to respond to a new alarm or other event may be handled in a more timely way. Collaboration between and contact with internal users, technicians, operators and/or engineers with known and recorded technical information and/or expertise may be carried out quickly and effectively. The engineer, operator or other user has a portable display device or portable computing and display device. This may comprise devices such as a PDA, as a head-up display arranged as glasses or on an adapted helmet. Preferably a camera or other position tracking means is mounted on or near the glasses, so capturing quite closely the user's view of the equipment, including the user's position and orientation, etc. Communication is preferably carried out using a head-set, mobile phone, phone and hands-free type attachment so that the user may communicate with other users, experts, and at the same time have one or more hands free for manipulating the display and or making an adjustment to a control panel or component etc. of the equipment, plant or process.

Collaboration with other internal users known to have the technical information and/or expertise that has been determined to be relevant to the equipment is enabled. Where necessary external experts or consultants may also be quickly identified and/or contacted. Information already stored in one or more different places or databases may be brought together in one situation or meeting room or electronic document and information or annotations from internal or external experts added to the document to provide the information and or instructions

necessary to carry out the control, maintenance or eg

configuration task in question. Annotations may be made in real time to information, text or graphics displayed to each user and or expert collaborating over the issue. Annotations or

attachments in the form of text annotations, digital files, video clips, sound clips may be attached to a shared electronic document or notice board, shared display screen means, virtual white board etc. The information retrieved and/or developed during collaboration provides instruction to the user for monitoring, troubleshooting, configuring or repairing the equipment . The method and system are facilitated by one or more computer programs in the control system and a computer architecture with which each user may be configured in a database, log-in file, table or similar to record one or more types of technical expertise he/she is known to possess and one or more types of communication means to the expert. When a software entity representing a component part of the equipment, plant or process in the control system is selected, a second software entity may be activated to access and retrieve information, especially contact information, phone numbers, IP phone numbers,

workstation URLs, GUIDs etc, for all internal (or external) users with expertise on the given component.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described, by way of example only, with particular reference to the accompanying drawings in which:

FIGURE 1 shows a schematic diagram of an overview of a system to generate control and/or maintenance instructions for an

industrial device or process according to an embodiment of the invention.

FIGURE 2 is a schematic diagram overview of a system to generate control and/or maintenance instructions for an industrial device or process according to another embodiment of the invention. FIGURE 3 is a schematic diagram of a screen display showing a selection on a composite image provided to generate control and/or maintenance instructions for an industrial device or process according to an embodiment of the invention;

FIGURE 4 is a schematic diagram of a screen display showing a selection on a composite image provided to generate control and/or maintenance instructions for an industrial device or process according to another embodiment of the invention; FIGURE 5 is a flowchart for a method for an improved way to access, annotate and display information for generating control and/or maintenance instructions for an industrial device or process according to an embodiment of the invention;

FIGURE 6 is a flowchart for a method for an improved way to access, share, collaborate and display shared information for generating control and/or maintenance instructions for an industrial device or process according to another embodiment of the invention.

FIGURES 7-9 shows a schematic diagram of a view of another screen display showing how an annotation may be fixed relative a point on a first image, according to an embodiment of the invention.

FIGURE 10 shows a schematic diagram of a view of a screen display showing how an annotation may be displayed as fixed relative a point on a display screen.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Figure 1 shows an overview for an improved system according to an embodiment of the invention. It shows a user 1, a device or equipment 7 to be monitored, repaired or controlled, and a display 3. The user holds a visible pointer 4 of some sort and a wearable computer 2. The user has a camera 6 of some sort preferably arranged mounted so as to record an image from the position and orientation direction that the user is looking in. The user has the display device arranged in the form of a head- up display 3, glasses or video glasses. A schematic view as seen by user 1 of the composite image 5 provided in display device 3 is shown. Signal, data and image information is

communicated to the display device and/or the control system 12, databases 14, by the wearable computer 2 via a data network 10. The view seen by the user comprises an image, or a composite image, of a first image 7' of the real world equipment and a view 4' of the pointer 4 relative to the image 7' of the real world equipment. In addition any of a text 9a, a diagram 9b or an annotation 9c may be superimposed over the image 7' of the equipment. The composite image 5 displayed to user 1 may

comprise a combination of an image 7 ' of the equipment of interest, an image of some information in the form of an image or text 9a, and, a diagram or drawing 9b and an added text, sketch, graphic or diagram 9b that has been added to the other two images as an annotation or attachment .

Figure 2 shows schematically a user 1, a wearable computer 2, a first display device 3a and a second display device 3b. The equipment object 7, the device or process of interest, is shown arranged with a marker 8 for the purpose of identifying it. The user may have a camera integrated with the first display 3a worn by the user 1 as a monocular viewer, glasses or head-up display. The computing functions may be provided by the wearable computer 2, by the second or alternative display 3b, the glasses or head- up display 3a, and/or may be distributed among these devices.

Figure 3 shows schematically another embodiment of the improved system. The figure shows the composite image 5 seen by user 1 comprising a view of the equipment 7 of interest. The view shows an image 4' of the pointer 4 held by the user displayed relative the equipment. Information, retrieved information 9a, 9b and added annotations 9c from one or more other users or experts may each be superimposed on any part of the composite image of the equipment. The annotations may be configured or arranged to be fixed relative a part of the first view of the equipment. In this case, the annotations move and change size or even

disappear from the composite image on the display when the user turns his/her head or changes position. In this way, the annotations are always associated with a particular part of the equipment or of the image (see also description and Figures 7- 10) .

The annotations may otherwise configured or arranged. For example the annotations may be in a fixed position relative a part of the computer display frame or apparatus screen. In this case, the or certain of the annotations stay on screen

independent of how the user changes his/her orientation or position. In this way, particular critical or selected

annotations are always available for the user to see. For example a virtual control panel or an on-line parameter reading may be continuously displayed even when the user moves his head and looks at other views .

Referring again to Figure 3. In the composite image 5 displayed to the user a virtual control panel 30 for the equipment 7 as identified to the control system by, for example, image

processing a unique marker 8, may be retrieved from data storage accessible by the control system, generated, and provided superimposed as an annotation on the image 7' of the equipment. When the user look at the real world equipment, the user sees the virtual control panel (VCP) as if it were attached to the real world equipment 7 because his head position is tracked by AR tracking means such as camera 6 and the image recorded and displayed to the user is adjusted accordingly. In this case the VCP may be arranged always fixed to a selected part of the equipment image, or may be arranged visible on the display while the user orientation and position change.

The virtual control panel 30 comprises a plurality of input or control elements, which may include a mode or status indicator 33, part name or number 34, one or more parameters or set-points 35, selection or activity indicators 88 and control means or control buttons 37. In the figure it is schematically shown that the virtual control panel may have a Status or Mode set to On Line, such that real time values for parameters such as % load, case temperature, rotational speed in rpm and supply current in amps etc are displayed. Other Modes may be selected such as for example Modes such as Off Line, On Line or Trend. Mode Trend provides for data retrieval from the control system and presents historical data or calculates predicted data or trends

respectively for the equipment of interest.

The user 1 may monitor and inspect a condition of an equipment 7 using an embodiment of the improved system. The user selects an equipment, which is identified in the control system by means of a marker 8 or alternative means, for example scanning methods as described below. User 1 may then retrieve information relevant to the identified equipment by means of the control system 12, stored in different formats in one or more databases 14, and examine the information. The user, an online expert or another person may record annotations by voice or text or freehand notes and attach those annotations 9c.

Other selected local or remote users or local or remote experts may share the composite image seen by the user 1 for the purpose of collaboration. The other users may add information in the form of notes or voice recordings or video clips which are attached to the composite image and shared by all selected users. Figure 3 shows schematically attached text information 9a, attached freehand notation 9b, attached diagram or flowchart 9c. The attached information which may have been contributed by other experts or users, called here annotations, may be used by user 1 as instructions or additional to carry out actions such as: carry out an inspection, adjust a set point, control a device or process, switch a device on or off. These actions are carried out by the user 1 by means of switching or operating buttons 37 or selection means 38 on the virtual control panel 30. This done by the user 1 moving the image 4' of pointer 4 viewed in the composite image 5 to select those buttons, by the action of moving the actual pointer 4 in the real world. As shown in Figure 3 a user may then inspect a condition of a motor and access or retrieve examine real time values for parameters such as speed, temperature, and load. Alternatively the user may make control inputs and or actions using a known industrial control or process control system to input set-points, switch a device on or off etc..

Thus in an aspect a method of an embodiment of the invention may comprise that the user 1 receives technical information or additional technical information from a local or internal colleague relevant to a device, apparatus or problem that the person is trained for or responsible for.

The method according to an embodiment of the invention comprises the steps of that a maintenance person requiring technical information picks up or preferably puts on a user AR equipment, which comprise be a PDA, wearable computer, headset based device, comprising a display 3, 3a, 3b. To deal with an incoming alarm, the control system 12 may generate information instructing the logged on maintenance person where in the plant to go and what the problem is (the system may indicate a new alarm via text, a visual or graphic display, or text-to- speech (TTS) . The maintenance person or user can ask for more info via text or voice. The maintenance person goes to a physical location and or a location of a plant or process section (functional location) indicated by the system, observes the problem, alarm or other event. The AR equipment may be used to recognise an equipment of interest 7 by means of: an asset number or similar, a wireless node mechanism, a marker 8 installed for the purpose of identifying the equipment to an AR system, a method of scanning equipment surroundings and processing the image to identify natural features to match to a predetermined location of the equipment, or a combination of equipment identity in the incoming alarm information together with any of the above. When the equipment of interest has been identified by one or other method to the control system the maintenance person or user can then access stored and/or real time information associated with the equipment through pre- configured associations of the control system.

The maintenance person can view service history and

documentation, such as for example:

-lists with instructions from the supplier of the particular object,

-top 5 problems for this particular device or item,

-view or listen to user or other operator notes recorded at an earlier date by a maintenance person or other operator who has fixed problems in the past (may even be his own notes) . The user can enter a new text or record a voice operator note or record a sound from the equipment etc regarding the problem.

If the maintenance person cannot fix the problem on his own, and needs help/support, he activates or mouse-clicks software means in the control system associated with the device or problem of interest. The system finds an appropriate online expert (where navigation options may include using a history of previous similar problems or a maintenance history of the selected device, process) or the user may select a specific person, and contact is then established by computer and/or by phone in a suitable way.

The information stored on the specific person or expert includes a unique address, internally with IP addresses and workstation addresses of users internal to the plan, and, preferably for external experts an IP address, URL, phone number etc of a support database, preferably an ABB database, and information about the local system to be used in support requests.

The second person agrees to accept the support request and receives access to the alarm or event, and gets an overview of at least the latest event as a starting point. Voice contact may be established automatically, for example via voice-over-IP (VoIP) . Options are provided for application sharing and/or to send pictures or files, and in particular for the selected specific person to share the composite image 5 seen by the user, and to add notes, comments, other information which are attached to the composite image as annotations. The second person, depending on user log-in information and/or authority or

privilege data, be enabled to also control the on-screen pointer image 4' by any computer input means suitable for selecting a control element of the VCP displayed by a graphical user

interface.

Annotations may include the possibility of attaching video, voice or pictures to a support request, and retrieving a "buddy- list" to give an overview of available local experts (Figure 5) . It may be desirable to add some logic to the list so that local contacts with knowledge about a specific problem are given priority, and so that the logic establishes which users or experts are now available, logged in, on call, suitable but not available, when available and why. An operational difference exists between local experts and remote (eg ABB Support)

experts. This is that support from remote experts is most often asynchronous at first, whilst contact with a local expert is typically synchronous from the beginning.

Figure 5 shows a first flowchart for a method which comprises the following steps: 51 User is notified by control system 12 in some way of an alarm or event

52 User takes AR inspection system, goes to the process, and an equipment of interest is identified to the control system

53 User examines equipment and the AR system displays a

composite image 5 of the real equipment combined with AR

information

55 Information such as documentation, annotations recorded in sound, picture or video form, a virtual control panel and so on is retrieved by software means of the control system 12 from databases such as 14,

56 User examines the retrieved information, may request other information, and decides if the present alarm or event situation is already known or described

59 If already described the user arranges for a repair of adjustment or replacement etc,

57 If not already described, records annotations of status, parameters, set points or other data by sound recording, text input, freehand sketch and so on,

58 User may retrieve and examine a list of internal operators, experts, and/or external experts with any of whom the user may choose to seek information or instructions for controlling an equipment or process .

Figure 6 shows a second flowchart for a method which comprises the following steps:

61 User chooses one or more internal and/or external experts

63 Request communicated to chosen internal and/or external expert contact means, IP phone, video phone, URL etc addresses 64 Expert (s) respond or agree via messaging system, SMS, e-mail, phone etc to help 65 Internal and/or external expert receives access to latest and all event details, access to related information via the control system, including latest annotations 57 added by user,

67 User and expert (s) discuss available technical inspection data, status and other information in the composite image 5, retrieve other information, exchange information and so on,

68 User and expert (s) record equipment configuration changes or set point changes, inspection data, pictures, sounds, voice, text or handwriting via a suitable input touch screen or stylus etc device,

69 After documenting technical data user arranges a fix or adjustment by actions such as switching on or off equipment or generating a control signal or changing one or more set-points via the virtual control panel 30 or via the control system 12, or making changes via a control unit physically at the location, and/or arrange for other maintenance actions, or repair or replacement as necessary.

Figure 4 shows another embodiment of the improved system. The figure shows another composite image 5 seen by user 1 comprising a view of the equipment 7 of interest. The view shows an image 4' of the pointer 4 held by the user displayed relative the equipment. Information, retrieved information 9a, 9b and added annotations 9c from one or more other users or experts may each be superimposed on the composite image of the equipment. The view of the equipment 7 of interest includes a unique marker 8 positioned in the view and, in this case, positioned on the actual equipment itself. The marker may be identified by using an image processing software to detect presence of a marker and identity of the marker on the equipment of interest.

In the composite image 5 displayed to the user a virtual control panel 30 for the equipment 7 as in the control system identified by marker 8 may be retrieved from data storage, generated, and provided as superimposed on the image 7' of the equipment. As described above in relation to Figure 3, the virtual control panel 30 comprises a plurality of control elements, which may include a mode or status indicator 33, part name or number 34, one or more parameters or set-points 35, selection or activity indicators 88 and control means or control buttons 37. In the figure it is schematically shown that the virtual control panel may have a Status or Mode set to Edit, such that set-point values for motor parameters such as: case temperature,

rotational speed in rpm and supply current in amps .

A collaborative session between the user and other local or remote engineers, specialists or experts may result in

information produced and attached to the composite image 5. The figure also represents an example of such method steps, in that a first 9b annotation was provided giving instructions, in this case for the user to "select Edit" mode, and a dotted

representation of the pointer image 4' applied to a control element of the VCP to select that Edit mode. Secondly another annotation 9b is made instructing the user to "increase RPM to 200" ie for the user to change the motor speed set-point and increase from 150 to 200 rpm. Pointer image 4' with solid lines indicates pointer image 4' positioned on a control element and activating the control element.

Other Modes may be selected such as for example Modes such as Off Line or Trend which retrieves from the control system and presents historical data or predicted data respectively for the equipment .

The virtual or augmented information is annotated and linked to the first view 7 of the equipment. Up to now the equipment has been described as identified in this case by a unique marker 8 arranged on or near the equipment, which is then matched with stored data about process devices, equipment etc.. Thus as shown in Figures 7-9 an annotation 9d is associated with part of of a control panel in a first image 7 of the equipment. In figure 8 the view 17 of the equipment has changed, but the annotation is still fixed to the same part of the equipment image, despite that the view or orientation of the user has changed. Similarly in Figure 9 the equipment view 27 is

different from first view 7, but the annotation is still

associated with the selected equipment part, so that the user sees the annotation whenever the equipment shown in the first view 7 is visible to him. If the user looks way the annotation vanishes. If the user looks back to the equipment the annotation re-appears because it is associated or fixed to a selected part of the first image. Figure 10 illustrates an annotation form

Figure 8 arranged fixed relative the screen display, so that as the user changes orientation and position the annotation remains in view in the same place.

In another embodiment the annotation 9 in the augmented reality view of the composite image equipment 7', I'¹ may not be associated to a selected part of the equipment or to any part of the displayed image and may instead "float" or remain in view indefinitely even when the user changes position and orientation completely. Thus an annotation may remain on the display even when the user goes to another part of the plant, if the user so desires .

In another embodiment the unique marker is identified by means other than by image processing to recognise a sign, symbol, number or other visual marker. Non-visual markers such as IR readable signs, or markers using other means such as acoustic transmissions may be adapted for use. Equipment identity may also be established using one or more purpose-specific wireless transmitters in the vicinity of the equipment of interest, or by using wireless communications based on wireless transmitters forming part of a data network in the vicinity.

In another embodiment the equipment is identified for tracking purposes and tracked relative a view of the real world using an optical mapping technique that involves identifying a series of objects or features in two or more repeated images taken by a camera or video camera of the equipment and its surroundings. An image processing software is used to identify objects detected optically, track the same objects over a succession of images, and calculate a position and orientation of the camera or sensor receiving and taking the images. In this way, the position and orientation of the user with a tracking apparatus 6, figures 1, 2 can be calculated. The image processing may be based on recognising and natural features, especially features with relatively higher contrast to their background, and subsequently calculating, based on the change in a distance between those natural features in an image, a position of the viewer relative the equipment or other object. Changes in the position and size of the annotation relative the equipment due to any change in position or orientation of the user's head are calculated in this way and the size and position of the annotation relative the equipment (see Figures 7-9) is displayed accordingly.

The image processing may be based in part on a Natural Features method. This type of method concerns identifying points on an object or in a surrounding, points such as edges or corners that show some visual contrast. As the scanner or camera moves orientation and point of view an image processing program tracks identified points in one or more successive frames. From these identified points and the paths the points move along the position and orientation of the scanner or camera is calculated. This position data is in turn used to determine, based on position or location data stored for the various process devices, equipment in the control system, which equipment in the plant is being looked at or inspected etc on the basis of the position so estimated. By this means the equipment can be identified without the use of a identifying marker when stored position data for devices, equipment at the location are available

In another embodiment the selected part of the equipment of interest is identified and tracked in a view of the real world by measuring a distance to one or more wireless node means arranged on or near said equipment. Transmissions from one or more wireless nodes comprised, for example, in a wireless LAN or in an ad-hoc wireless network, in a Bluetooth pico-net or a ZigBee network may be used to calculate a relative physical location based on signal strength of other radio signal

characteristics. From measurements of relative distance the equipment may be mapped to location data stored in a control system of the industrial plant.

The methods of the invention may be carried out by means of one or more computer programs comprising computer program code or software portions running on a computer, data server or a processor. The microprocessor (or processors) comprises a central processing unit CPU performing the steps of the method according to one or more facets of the invention, such as the methods shown in Figures 5, 6. The methods are performed with the aid of one or more said computer programs, which are stored at least in part in memory accessible by the one or more processors. For example a program or part-program that carries out some or all of the steps shown and described in relation to in Figures 5 and 6 may be run by a computer or processor of the control system. At least one of the or each processors may be in a central object oriented control system in a local or distributed computerised control system. It is to be understood that said computer programs may also be run on one or more general purpose industrial microprocessors or computers instead of one or more specially adapted computers or processors .

The computer program comprises computer program code elements or software code portions that make the computer perform the method using equations, algorithms, data, stored values and

calculations previously described. A part of the program may be stored in a processor as above, but also in a ROM, RAM, PROM, EPROM, or EEPROM chip or similar memory means. The program in part or in whole may also be stored on, or in, other suitable computer readable medium such as a magnetic disk, CD-ROM or DVD disk, hard disk, magneto-optical memory storage means, in volatile memory, in flash memory, as firmware, stored on a data server or on one or more arrays of data servers . Other known and suitable media, including removable memory media such as removable flash memories, hard drives etc. may also be used.

Data may also be communicated wirelessly, at least in part, to portable devices carried or worn by a user. Wireless

communications may be carried out using any suitable protocol, including a wireless telephone system such as GSM or GPRS.

Short range radio communication is a preferred technology, using a protocol compatible with, standards issued by the Bluetooth Special Interest Group (SIG), any variation of IEEE-802.11, WiFi, WiMax, Ultra Wide Band (UWB), ZigBee or IEEE-802.15.4, IEEE-802.13 or equivalent or similar. Wireless communication may also be carried out using Infra Red (IR) means and protocols such as IrDA, IrCOMM or similar. Wireless communication may also be carried out using sound or ultrasound transducers, through the air or via work object construction, pure magnetic or electric fields (capacitive or inductive communication) or other types of light, such as for example LED, laser, as communication media with standard or proprietary protocols.

The computer programs described above may also be arranged in part as a distributed application capable of running on several different computers or computer systems at more or less the same time. Programs as well as data such as energy related

information may each be made available for retrieval, delivery or, in the case of programs, execution over the Internet. Data and/or methods may be accessed by software entities or other means of the control system by means of any of the lost of: OPC, OPC servers, an object request broker such as COM, DCOM, or CORBA, or a web service.

It is also noted that while the above describes exemplifying embodiments of the invention, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention as defined in the appended claims .

Claims

1. Method for processing and displaying control or maintenance instructions for an apparatus, device or equipment (7) in an industrial process or facility comprising one or more control systems (12) for monitoring and control, wherein device or process-related information and other data for each said

equipment, plant or process are stored and may be retrieved by said control system, characterised by

-identifying a said equipment (7) and making and processing an image of a first view (7') of the said equipment,

-making, retrieving or recording an annotation (9, 9a-c) of control or maintenance instructions for said equipment and arranging the annotation represented as an active or passive graphic element for a display,

-combining the first view of the image and the annotation into a composite image (5) , and

-displaying the composite image of said equipment combined with additional control or maintenance instructions to a user.

2. A method according to claim 1, wherein the annotation for said equipment is placed in the display at a selected position relative to a part of the first view of the image.

3. A method according to claim 2, wherein the annotation is generated as a part of a layer for superposition on the first view of the image.

4. A method according to claim 2 or 3 , wherein the annotation is placed in the display at a selected position relative to a position on the display screen.

5. A method according to claim 1, wherein a display of an instance of the composite image is displayed on a second display to a second user or expert.

6. A method according to claim 5 , wherein an annotation is provided by an action of a second user or expert manipulating a second instance of the composite image displayed on the second display.

7. A method according to claim 6, wherein data relevant to said equipment in the form of one or more annotations is added to the composite image and displayed to a plurality of other users.

8. A method according to any of claims 1-7, wherein part (30, 4', 37) of the composite image (5) comprises software means for generating control or maintenance instructions and is arranged with means to configure a set point, to generate a control signal or to switch a device on or off, in response to user activation (4' ) .

9. A method according to claim 1, wherein said equipment is identified by unique marker means (8) on or near said equipment.

10. A method according to claim 9, wherein said equipment is identified in part by applying an image processing software to the unique marker means (8) .

11. A method according to claim 1, wherein said equipment is identified in part from images produced by optical means (6) .

12. A method according to claim 11, wherein said equipment is identified in part by applying an image processing software to the images provided by the optical means.

13. A method according to claim 12 , wherein the images are processed in part by scanning natural features of said

equipment, other objects and/or surroundings of said equipment.

14. A method according to claim 13 , wherein the images are processed in part by calculating a movement path of the

camera/viewer based on changes calculated in a viewer-to-object distance for one or more natural features of said equipment, other objects and/or surroundings of said equipment.

15. A method according to any of claims 1-14, wherein a size relationship between two or more parts (V, 4') of the first view of the image is determined and used to calculate a position and or orientation of a user (1) .

16. A method according to claim 15, wherein the position and/or orientation of the annotation relative a second view of the image is changed dependent on the distance relationship or a relative size relationship determined with the first view.

17. A method according to claim 16, wherein either or both the position and size of the annotation are changed depending on a calculating image part distance relationships in the second view of the image dependent on the distance relationship between parts of the image, or a relative size relationship, determined for the first view.

18. A method according to any of claims 15-17, wherein a movement path of the user gaze and orientation is calculated during present and successive views depending on the distance or size relationship of parts of the image with each other

determined for a previous view.

19. A method according to claim 1, wherein said equipment is identified by processing transmissions from one or more wireless node means arranged on or near said equipment .

20. A method according to any previous claim, characterised by generating instructions and information for carrying out any member of the group consisting of: repair, re-configure, re- programming or replacement of a faulty part of said equipment, plant or process .

21. A computer program for processing and displaying control or maintenance instructions for an apparatus, device or equipment

(7) in an industrial process or facility comprising one or more control systems (12) for monitoring and control, comprising computer code means and/or software code portions which when run on a computer or processor will make said computer or processor perform the steps of a method according any of claims 1-20.

22. A computer program comprising a computer program according to claim 21 comprised in one or more computer readable media.

23. A system for processing and displaying control or

maintenance instructions for an apparatus, device or equipment in an industrial process or facility comprising one or more control systems (12) for monitoring and control, and one or more computers, wherein device or process-related information and other data for each said equipment, plant or process are stored and may be retrieved from said control system,

characterised by

-an AR system comprising a tracking means (6) , a display means (3, 3a, 3b) at least one computer means (2), a data

communication means (10) for communicating an image (7') of a first view (7) of said equipment,

-computer software image processing means for identifying the said equipment in the industrial process or facility,

-computer software means to record control or maintenance instructions comprising any of text, video or voice messages and computer software for attaching said message information, and

-computer software means to provide a composite image of an image of said equipment combined with additional control or maintenance instructions to a user.

24. A system according to claim 23, wherein at least one of augmented reality system comprises a handheld interacting and pointing device (4) .

25. A system according to claim 24, wherein the tracking system (6) comprises means for determining position and orientation of the handheld device.

26. A system according to claim 25, wherein the tracking system (6) comprises means for determining position and orientation of the handheld device based on one or more identifying markers (8) .

27. A system according to claim 25, wherein the tracking system comprises means for determining position and orientation of the handheld device based on processing an image comprising one or more identifying markers (8) .

28. A system according to claim 25, wherein the tracking system comprises means for determining position and orientation of the handheld device based on identifying one or more natural features in the view of said equipment or surroundings thereto.

29. A system according to claim 25, wherein the tracking system comprises means for processing an image to determine position and orientation of the handheld device based on spatial

relationships between the one or more natural features in the view of said equipment, other objects or surroundings thereto and with the handheld device.

30. A system according to claim 29, wherein the one or more visible features in the view of said equipment or surroundings comprise one or more edges or corners.

31. A system according to claim 23, wherein the system comprises a portable or wearable display device for visualizing a composite image comprising augmented reality overlaid on a real world view, which may be any form the list of: a monocular device, glasses, goggles, head-mounted display, helmet mounted display, head-up display.

32. A system according to any of claims 23-31, wherein the system comprises a data communication means (10) comprises a data network or LAN which may include one or more wireless nodes.

33. A system according to claim 32, wherein the system comprises a computer software means for arranging a position of an

annotation (9, 9a, 9b, 9c) relative part of an image or relative part of a display means of a display device (3, 3a, 3b) .

34. A system according to claim 23, characterised by comprising computer program means according to any of claims 21, 22.

35. Use of a system for maintenance and/or control in an industrial process or device according to any of claims 24-28 for providing control or maintenance instructions in response to an alarm or event in a plant or process in a facility comprising a plurality of devices and one or more control systems for process monitoring and control .