US7296239B2 - System GUI for identification and synchronized display of object-correspondence in CT volume image sets - Google Patents

System GUI for identification and synchronized display of object-correspondence in CT volume image sets Download PDF

Info

Publication number
US7296239B2
US7296239B2 US10/090,314 US9031402A US7296239B2 US 7296239 B2 US7296239 B2 US 7296239B2 US 9031402 A US9031402 A US 9031402A US 7296239 B2 US7296239 B2 US 7296239B2
Authority
US
United States
Prior art keywords
interest
location
image sets
displaying
displays
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US10/090,314
Other versions
US20030164860A1 (en
Inventor
Hong Shen
Jianzhong Qian
Li Fan
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens AG
Original Assignee
Siemens Corporate Research Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Corporate Research Inc filed Critical Siemens Corporate Research Inc
Priority to US10/090,314 priority Critical patent/US7296239B2/en
Assigned to SIEMENS CORPORATE RESEARCH, INC. reassignment SIEMENS CORPORATE RESEARCH, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SHEN, HONG
Assigned to SIEMENS CORPORATE RESEARCH, INC. reassignment SIEMENS CORPORATE RESEARCH, INC. RE-RECORD TO ADD ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL/FRAME 013020/0338 Assignors: FAN, LI, QIAN, JIANZHONG, SHEN, HONG
Priority to CNA038052431A priority patent/CN1639739A/en
Priority to PCT/US2003/006395 priority patent/WO2003077203A2/en
Priority to JP2003575341A priority patent/JP2005518916A/en
Priority to DE10392341T priority patent/DE10392341T5/en
Publication of US20030164860A1 publication Critical patent/US20030164860A1/en
Publication of US7296239B2 publication Critical patent/US7296239B2/en
Application granted granted Critical
Assigned to SIEMENS CORPORATION reassignment SIEMENS CORPORATION MERGER (SEE DOCUMENT FOR DETAILS). Assignors: SIEMENS CORPORATE RESEARCH, INC.
Assigned to SIEMENS AKTIENGESELLSCHAFT reassignment SIEMENS AKTIENGESELLSCHAFT ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: SIEMENS CORPORATION
Adjusted expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects

Definitions

  • This invention relates to a graphical user interface for object-correspondence identification between computed tomography (CT) volume data sets in general, and to lung nodule-correspondence identification between CT volume data sets in particular.
  • CT computed tomography
  • Lung computed tomography (CT) technology has been widely used by physicians in the screening and diagnosis of lung cancer. From lung CT images, the physician can search for nodules and make judgments on their malignancy based on the statistics of the nodules, including shape, size, etc. A very important piece of information is the status change of the nodules over time, such as changes in shape, size, and density.
  • One of the most significant quantitative measurements is the growth rate of lung nodules during a period of time. It is therefore crucial to identify the correspondence of the same nodule in two or more lung CT image sets captured at different time frames.
  • the CT data are 3D images
  • the task becomes very difficult for the physician, if at all achievable.
  • the physician is required to scan through 2D slices of the 3D image data one by one and try to find the correspondence of a nodule in two image sets.
  • the number of slices for a single data set is as large as several hundreds, and a single slice contains 250,000 pixels.
  • the imaging condition for the patient in the CT studies may be varied and the organ and the body may be deformed between two such studies. In many cases, it is hard to tell if a nodule has disappeared after a period of time or still exists because the physician is not able to identify the correspondence between images.
  • VOI local volumes-of-interest
  • medical image analysis systems such as the systems for analyzing lung CT images.
  • VOI local volumes-of-interest
  • very important pieces of information are the presence of a new nodule, the absence of a previously presented nodule, and the growth rate of a lung nodule. It is therefore crucial to identify the correspondence of the same nodule in two or more lung CT image sets captured at different time frames.
  • the properties of the nodule and its surrounding structures are locally distinct, and therefore the registration of local VOI's is sufficient for identifying the correspondence of nodules.
  • GUI graphical user interface
  • GUI graphical user interface
  • main displays each for displaying an image set
  • user interactive system for receiving from a user a first location of an object-of-interest in one of said image sets
  • correlation system for finding and displaying a second location, corresponding to said first location, of a corresponding second volume-of-interest in at least one of said other image sets.
  • a first volume-of-interest is defined about said first location of said object-of-interest.
  • said first and second volumes-of-interest are rendered as Shaded Surface Displays.
  • said image sets are rendered as Shaded Surface Displays.
  • GUI further comprises displays permitting the user to examine said volumes-of-interest with free viewpoints.
  • GUI further comprises one or more data windows for displaying image properties of at least one of said image sets.
  • GUI further comprises a lock-scrolling system for synchronized scrolling of two or more image sets.
  • GUI further comprises a cartwheel projection system for side-by-side display of two or more cartwheel projection spin windows.
  • GUI further comprises a first property display for displaying physical properties of said object-of-interest, a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest, and wherein said first and second property displays may be displayed side-by-side.
  • a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for a graphical user interface for an object-correspondence system, said method steps comprising providing a plurality of main displays, each for displaying an image set, receiving from a user a first location of an object-of-interest in one of said image sets, and receiving a second location of a second volume-of-interest from a correlation system, said second location corresponding to said first location.
  • a first volume-of-interest is defined about said first location of said object-of-interest.
  • said first and second volumes-of-interest are rendered as Shaded Surface Displays.
  • said image sets are rendered as Shaded Surface Displays.
  • Another aspect of the storage device further comprises the function to allow the user to examine said volumes-of-interest with free viewpoints in 3D in synchronization.
  • Another aspect of the storage device further comprises one or more data windows for displaying image properties of a volume-of-interest at least one of said image sets.
  • Another aspect of the storage device further comprises machine-readable code for synchronized scrolling of two or more image sets.
  • Another aspect of the storage device further comprises machine-readable code for side-by-side display of two or more cartwheel projection spin windows.
  • Another aspect of the storage device further comprises machine readable code for providing a first property display for displaying physical properties of said object-of-interest, providing a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest, and displaying said first and second property displays side-by-side.
  • a method of interfacing graphically with a user for an object-correspondence identification system comprising the steps of providing a plurality of main displays, each for displaying an image set, receiving from a user a first location of an object-of-interest in one of said image sets, and receiving a second location of a second volume-of-interest from a correlation system, said second location corresponding to said first location.
  • a first volume-of-interest is defined about said first location of said object-of-interest.
  • said first and second volumes-of-interest are rendered as Shaded Surface Displays.
  • said volumes-of-interest are rendered as Shaded Surface Displays.
  • Another aspect of the method of the invention further comprises providing the function to allow the user to examine said volumes-of-interest with free viewpoint in 3D in synchronization.
  • Another aspect of the method of the invention further comprises one or more data windows for displaying image properties of at least one of said image sets.
  • Another aspect of the method of the invention further comprises providing a lock-scrolling system for synchronized scrolling of two or more image sets.
  • Another aspect of the method of the invention further comprises providing a cartwheel projection system for side-by-side display of two or more cartwheel projection spin windows.
  • Another aspect of the method of the invention further comprises providing a first property display for displaying physical properties of said object-of-interest, providing a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest, and displaying said first and second property displays side-by-side.
  • FIG. 1 shows an embodiment of the graphical user interface of the invention.
  • FIG. 2 shows SSD of illustrative detected correspondences.
  • FIG. 3 shows a flowchart of an embodiment of the invention.
  • GUI graphical user interface
  • the physician can use this invention to click on an object in one image set, and the system defines a volume-of-interest around this click point, and then identifies a corresponding volume-of-interest in the other image sets automatically.
  • the identification is between two volumes-of-interest abundant with anatomy structures, the user can easily identify the correspondence of objects in these volumes, or confidently decide that an object in the corresponding volume-of-interest has disappeared over the time between two computed tomography (CT) scans.
  • CT computed tomography
  • the invention is particularly useful for medical imaging, such as tracking lung nodules.
  • Two scans of a patient's lung may be shown side-by-side in the GUI, and when the user clicks on a nodule on one image set, the system then automatically identifies the corresponding location in the remaining image set. The physician can then immediately see whether the nodule is changing, migrating, or has simply vanished.
  • the nodules-of-interest with measurements of their properties are also displayed side-by-side.
  • a new GUI 10 is disclosed.
  • the novel GUI 10 is based upon a GUI design from two previously filed commonly assigned U.S. patent applications, namely Novak et al., Interactive Computer-Aided Diagnosis Method and System for Assisting Diagnosis of Lung Nodules in Digital Volumetric Medical Image sets, U.S. Ser. No. 09/840,267, filed Apr. 23, 200 1, and Novak et al., Computer-Aided Diagnosis Method for Aiding Diagnosis of Three Dimensional Digital Image Data, U.S. Ser. No. 09/606,564, filed Jun. 29, 2000, the disclosures of both of which are incorporated by reference herein in their entirety.
  • the GUI disclosed in these patents allows the displaying, scrolling, spinning, and Surface Shaded Display (SSD) rendering of the CT lung image sets.
  • SSD Surface Shaded Display
  • two or more image sets 20 for the same patient are displayed in main display windows 15 for comparison, each representing a patient's lung at different times.
  • the image sets will preferably be three dimensional, meaning the user can scroll through slices of each image set by interactive means, such as by the pressing of the middle button of a mouse and moving the mouse, or other known GUI interactive means.
  • the user can also select an object-of-interest 30 on any image set, and perform spinning 50 or surface rendering 60 on a volume-of-interest (VOI) 40 centered upon the object-of-interest.
  • VOI volume-of-interest
  • the GUI of the invention will provide additional functionality, such as allowing the user to select a lock-scrolling mode.
  • this mode when the user scrolls through the slices of one image set, the corresponding slices of the other image set(s) are displayed in the other display window(s).
  • this invention provides a system for synchronized scrolling of the slices amongst image sets.
  • the invention will preferably also comprise a system to allow two or more cartwheel projection spin windows 50 to be simultaneously displayed side by side, one in each image set, so that the user can adjust the corresponding projection angles of two volumes-of-interest (VOI's). This helps the user to visually compare structures in the two VOI's.
  • VOI's volumes-of-interest
  • various displays 70 may be provided upon the screen to indicate useful parameters and measurements, such as the current location of objects-of-interest, the sizes of the objects, growth rate.
  • property displays 70 are shown allowing the user to compare the physical properties of the two objects-of-interest side-by-side.
  • Various controls 80 may be provided to control the slice depth, the rendering of VOI's with free viewpoints, and other displaying features.
  • free viewpoints is meant that the 3D image may be rotated freely, permitting the VOI's to be viewed from any viewpoint, the number of viewpoints being substantially unlimited.
  • the GUI 10 of the invention possesses a novel core functionality, namely that the selection of an object-of-interest in one image set causes the system to define a volume-of-interest in the remaining image set centered about the location that corresponds to that of the object-of-interest.
  • the user has selected a nodule 30 and a user interactive sub-system has received the user's input and marked out a first volume-of-interest 40 a associated with this nodule.
  • the corresponding second volume-of-interest 40 b is then automatically identified and marked on the other image set in the upper-right display window by a correspondence identification sub-system.
  • the system will preferably also allow the user to examine the two SSD's with free viewpoints in synchronization to visually compare the properties of the two nodules and VOI's.
  • the surrounding structures will preferably also be displayed to show the validity of the correspondence.
  • the geometric properties of the objects will preferably be displayed in data windows 70 for comparison.
  • the system can easily compute the growth rate of the nodule over time. If a nodule does not exist in one volume-of-interest, the physician can confidently make the conclusion that it has disappeared rather than not been detected because the surrounding structures show a correct match between the two volumes-of-interest 40 .
  • the correspondence matching will preferably be able to be done in both directions. That is, the user can also click on and select a nodule in the second image set 20 b and a volume-of-interest will automatically appear in the first image set 20 a.
  • FIG. 2 there is shown online-detected correspondences rendered in SSD.
  • Three different cases are shown.
  • two corresponding nodules 130 are displayed and both are preferably marked in some fashion, such as by a color or by patterning, such as with diagonal lines as shown.
  • a growth rate of the nodule can be computed from the volumes of the two corresponding nodules and the time interval.
  • the second set of displays 110 two corresponding objects 30 , 30 ′ are detected, but one of them 30 ′ is marked differently, such as by another color or pattern, such as the vertical lines as shown. This indicates to the user that the software has determined that the second object 30 ′ does not meet the criterion of nodules.
  • the object-of-interest 30 has vanished from the VOI on the right. From the surrounding structures we can clearly see the correct correspondence between two VOI's, and we can conclude that the nodule is absent after a period of time.
  • FIG. 3 is a flowchart of an embodiment of the invention showing how the command flow of the GUI may be achieved, though of course there are many possible variants of command flow in the programming art.
  • the image sets are loaded at 200 . This step of loading the image sets could alternatively have been executed from box 240 as a file operation.
  • the GUI awaits a user command, which may be (1) the clicking on an object-of-interest, (2) a user input for change of display, or (3) a menu option. Box 210 would generally be event driven. The remaining boxes show the flow of execution of commands and return to box 210 as has been described above.
  • flow goes to box 220 where the correlation system is invoked to locate a corresponding VOI in a second image set.
  • a first VOI is also defined about the first object-of-interest.
  • Flow then goes to box 230 where the VOI's and object properties are displayed before control flow returns to the user at box 210 .
  • any one or more of the display changes described above are executed in real time at box 250 before control returns to the user.
  • any one of standard or specialized menu operations are executed at box 240 , including such familiar operations as file operations (e.g., open file, close file, save file, etc.), user and program preferences and settings, access to a help file, or simply exiting the program.
  • this invention would greatly ease and improve the quality of nodule correspondence identification and henceforth the screening and diagnosis of lung cancer. As the result, the clinical throughput and diagnostic accuracy would be greatly improved.
  • the methods of the invention may be implemented as a program of instructions, readable and executable by machine such as a computer, and tangibly embodied and stored upon a machine-readable medium such as a computer memory device.

Abstract

Disclosed is a graphical user interface, comprising a plurality of main displays, each for displaying an image set, a user interactive system for receiving from a user a first location of an object-of-interest in one of said image sets, and a correlation system for finding and displaying a second location, corresponding to said first location, of a corresponding second volume-of-interest in at least one of said other image sets.

Description

FIELD OF THE INVENTION
This invention relates to a graphical user interface for object-correspondence identification between computed tomography (CT) volume data sets in general, and to lung nodule-correspondence identification between CT volume data sets in particular.
BACKGROUND OF THE INVENTION
Lung computed tomography (CT) technology has been widely used by physicians in the screening and diagnosis of lung cancer. From lung CT images, the physician can search for nodules and make judgments on their malignancy based on the statistics of the nodules, including shape, size, etc. A very important piece of information is the status change of the nodules over time, such as changes in shape, size, and density. One of the most significant quantitative measurements is the growth rate of lung nodules during a period of time. It is therefore crucial to identify the correspondence of the same nodule in two or more lung CT image sets captured at different time frames.
So far, this task has been done manually, and therefore it is tedious, slow, and error prone because of the tremendous amount of data. Because the CT data are 3D images, the task becomes very difficult for the physician, if at all achievable. In current clinical practice, the physician is required to scan through 2D slices of the 3D image data one by one and try to find the correspondence of a nodule in two image sets. The number of slices for a single data set is as large as several hundreds, and a single slice contains 250,000 pixels. Moreover, the imaging condition for the patient in the CT studies may be varied and the organ and the body may be deformed between two such studies. In many cases, it is hard to tell if a nodule has disappeared after a period of time or still exists because the physician is not able to identify the correspondence between images.
Fast registration of local volumes-of-interest (VOI) from large 3D image data is very often needed in medical image analysis systems, such as the systems for analyzing lung CT images. For example, in the screening and diagnosis of lung cancer, very important pieces of information are the presence of a new nodule, the absence of a previously presented nodule, and the growth rate of a lung nodule. It is therefore crucial to identify the correspondence of the same nodule in two or more lung CT image sets captured at different time frames. In most cases, the properties of the nodule and its surrounding structures are locally distinct, and therefore the registration of local VOI's is sufficient for identifying the correspondence of nodules.
What is needed is a graphical user interface (GUI) that provides convenient examination of two or more image sets allowing the user to immediately identify an object-of-interest on one image set and thereby automatically identifies a corresponding region of interest on the remaining image set by virtue of an automated system that avoids full volume registration, but performs fast and accurate registration of two local VOI's. The GUI should also provide a various set of functions to facilitate examination and comparison, such as synchronized scrolling of the slices in the two data sets.
SUMMARY OF THE INVENTION
Disclosed is a graphical user interface (GUI), comprising a plurality of main displays, each for displaying an image set, a user interactive system for receiving from a user a first location of an object-of-interest in one of said image sets, and a correlation system for finding and displaying a second location, corresponding to said first location, of a corresponding second volume-of-interest in at least one of said other image sets.
In another aspect of the GUI a first volume-of-interest is defined about said first location of said object-of-interest.
In another aspect of the GUI said first and second volumes-of-interest are rendered as Shaded Surface Displays.
In another aspect of the GUI said image sets are rendered as Shaded Surface Displays.
Another aspect of the GUI further comprises displays permitting the user to examine said volumes-of-interest with free viewpoints.
Another aspect of the GUI further comprises one or more data windows for displaying image properties of at least one of said image sets.
Another aspect of the GUI further comprises a lock-scrolling system for synchronized scrolling of two or more image sets.
Another aspect of the GUI further comprises a cartwheel projection system for side-by-side display of two or more cartwheel projection spin windows.
Another aspect of the GUI further comprises a first property display for displaying physical properties of said object-of-interest, a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest, and wherein said first and second property displays may be displayed side-by-side.
Disclosed is a program storage device, readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for a graphical user interface for an object-correspondence system, said method steps comprising providing a plurality of main displays, each for displaying an image set, receiving from a user a first location of an object-of-interest in one of said image sets, and receiving a second location of a second volume-of-interest from a correlation system, said second location corresponding to said first location.
In another aspect of the storage device a first volume-of-interest is defined about said first location of said object-of-interest.
In another aspect of the storage device said first and second volumes-of-interest are rendered as Shaded Surface Displays.
In another aspect of the storage device said image sets are rendered as Shaded Surface Displays.
Another aspect of the storage device further comprises the function to allow the user to examine said volumes-of-interest with free viewpoints in 3D in synchronization.
Another aspect of the storage device further comprises one or more data windows for displaying image properties of a volume-of-interest at least one of said image sets.
Another aspect of the storage device further comprises machine-readable code for synchronized scrolling of two or more image sets.
Another aspect of the storage device further comprises machine-readable code for side-by-side display of two or more cartwheel projection spin windows.
Another aspect of the storage device further comprises machine readable code for providing a first property display for displaying physical properties of said object-of-interest, providing a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest, and displaying said first and second property displays side-by-side.
Disclosed is a method of interfacing graphically with a user for an object-correspondence identification system, comprising the steps of providing a plurality of main displays, each for displaying an image set, receiving from a user a first location of an object-of-interest in one of said image sets, and receiving a second location of a second volume-of-interest from a correlation system, said second location corresponding to said first location.
In another aspect of the method of the invention a first volume-of-interest is defined about said first location of said object-of-interest.
In another aspect of the method of the invention said first and second volumes-of-interest are rendered as Shaded Surface Displays.
In another aspect of the method of the invention said volumes-of-interest are rendered as Shaded Surface Displays.
Another aspect of the method of the invention further comprises providing the function to allow the user to examine said volumes-of-interest with free viewpoint in 3D in synchronization.
Another aspect of the method of the invention further comprises one or more data windows for displaying image properties of at least one of said image sets.
Another aspect of the method of the invention further comprises providing a lock-scrolling system for synchronized scrolling of two or more image sets.
Another aspect of the method of the invention further comprises providing a cartwheel projection system for side-by-side display of two or more cartwheel projection spin windows.
Another aspect of the method of the invention further comprises providing a first property display for displaying physical properties of said object-of-interest, providing a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest, and displaying said first and second property displays side-by-side.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows an embodiment of the graphical user interface of the invention.
FIG. 2 shows SSD of illustrative detected correspondences.
FIG. 3 shows a flowchart of an embodiment of the invention.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
In this invention the identification of object correspondence between two image sets is automated, which makes the procedure more accurate and objective. A graphical user interface (GUI) is provided that can display two or more volume image data sets simultaneously. The physician can use this invention to click on an object in one image set, and the system defines a volume-of-interest around this click point, and then identifies a corresponding volume-of-interest in the other image sets automatically. Because the identification is between two volumes-of-interest abundant with anatomy structures, the user can easily identify the correspondence of objects in these volumes, or confidently decide that an object in the corresponding volume-of-interest has disappeared over the time between two computed tomography (CT) scans. Second, the correspondence identification is real-time, that is, the correspondence of the volume-of-interest is done instantly at the click of the nodule.
The invention is particularly useful for medical imaging, such as tracking lung nodules. Two scans of a patient's lung may be shown side-by-side in the GUI, and when the user clicks on a nodule on one image set, the system then automatically identifies the corresponding location in the remaining image set. The physician can then immediately see whether the nodule is changing, migrating, or has simply vanished. Preferably, the nodules-of-interest with measurements of their properties are also displayed side-by-side.
Referring to FIG. 1, a new GUI 10 is disclosed. The novel GUI 10 is based upon a GUI design from two previously filed commonly assigned U.S. patent applications, namely Novak et al., Interactive Computer-Aided Diagnosis Method and System for Assisting Diagnosis of Lung Nodules in Digital Volumetric Medical Image sets, U.S. Ser. No. 09/840,267, filed Apr. 23, 200 1, and Novak et al., Computer-Aided Diagnosis Method for Aiding Diagnosis of Three Dimensional Digital Image Data, U.S. Ser. No. 09/606,564, filed Jun. 29, 2000, the disclosures of both of which are incorporated by reference herein in their entirety. The GUI disclosed in these patents allows the displaying, scrolling, spinning, and Surface Shaded Display (SSD) rendering of the CT lung image sets. By placing two such GUI's side-by-side and augmenting with now functionality, we construct the novel GUI of this invention.
Using medical imaging as an example, two or more image sets 20 for the same patient are displayed in main display windows 15 for comparison, each representing a patient's lung at different times. The image sets will preferably be three dimensional, meaning the user can scroll through slices of each image set by interactive means, such as by the pressing of the middle button of a mouse and moving the mouse, or other known GUI interactive means. The user can also select an object-of-interest 30 on any image set, and perform spinning 50 or surface rendering 60 on a volume-of-interest (VOI) 40 centered upon the object-of-interest. These functions are provided in the interactive computer-aided diagnosis (ICAD) patents cited above, but now are made available simultaneously for a plurality of image sets, rather than only one. In a preferred embodiment the spin window 50 will pop up when the spin functionality is selected by user, and can be dismissed by user input.
Preferably, the GUI of the invention will provide additional functionality, such as allowing the user to select a lock-scrolling mode. In this mode, when the user scrolls through the slices of one image set, the corresponding slices of the other image set(s) are displayed in the other display window(s). In other words, this invention provides a system for synchronized scrolling of the slices amongst image sets. The invention will preferably also comprise a system to allow two or more cartwheel projection spin windows 50 to be simultaneously displayed side by side, one in each image set, so that the user can adjust the corresponding projection angles of two volumes-of-interest (VOI's). This helps the user to visually compare structures in the two VOI's. Additionally, various displays 70 may be provided upon the screen to indicate useful parameters and measurements, such as the current location of objects-of-interest, the sizes of the objects, growth rate. Here, property displays 70 are shown allowing the user to compare the physical properties of the two objects-of-interest side-by-side. Various controls 80 may be provided to control the slice depth, the rendering of VOI's with free viewpoints, and other displaying features. By “free viewpoints” is meant that the 3D image may be rotated freely, permitting the VOI's to be viewed from any viewpoint, the number of viewpoints being substantially unlimited.
The GUI 10 of the invention possesses a novel core functionality, namely that the selection of an object-of-interest in one image set causes the system to define a volume-of-interest in the remaining image set centered about the location that corresponds to that of the object-of-interest. For example, referring again to FIG. 1, in the main left-hand display window 15 a, the user has selected a nodule 30 and a user interactive sub-system has received the user's input and marked out a first volume-of-interest 40 a associated with this nodule. The corresponding second volume-of-interest 40 b is then automatically identified and marked on the other image set in the upper-right display window by a correspondence identification sub-system. This is achieved by providing the GUI system of the invention with either the exact or estimated location of this position as is disclosed in commonly assigned U.S. patent application Shen et al., Object Correspondence Identification Without Full Volume Registration, Ser. No. 10/071,003 , filed Feb. 7, 2002, the disclosures of which are incorporated by reference herein in their entirety. The Shaded Surface Display (SSD) and measurements of two corresponding volumes-of-interest are rendered in the two lower display windows 60 a and 60 b side-by-side to facilitate user's comparison.
The system will preferably also allow the user to examine the two SSD's with free viewpoints in synchronization to visually compare the properties of the two nodules and VOI's. The surrounding structures will preferably also be displayed to show the validity of the correspondence. The geometric properties of the objects will preferably be displayed in data windows 70 for comparison. Hence, in the example shown where the object-of-interest is a lung nodule, by examining the geometric properties the system can easily compute the growth rate of the nodule over time. If a nodule does not exist in one volume-of-interest, the physician can confidently make the conclusion that it has disappeared rather than not been detected because the surrounding structures show a correct match between the two volumes-of-interest 40.
The correspondence matching will preferably be able to be done in both directions. That is, the user can also click on and select a nodule in the second image set 20 b and a volume-of-interest will automatically appear in the first image set 20 a.
Referring to FIG. 2, there is shown online-detected correspondences rendered in SSD. Three different cases are shown. In the top row of displays 100, two corresponding nodules 130 are displayed and both are preferably marked in some fashion, such as by a color or by patterning, such as with diagonal lines as shown. This indicates to the user that the software has determined that both objects-of-interest meet the criterion of the object sought after. In our lung example it means that the object meets the criterion of nodules. A growth rate of the nodule can be computed from the volumes of the two corresponding nodules and the time interval. In the second set of displays 110, two corresponding objects 30, 30′ are detected, but one of them 30′ is marked differently, such as by another color or pattern, such as the vertical lines as shown. This indicates to the user that the software has determined that the second object 30′ does not meet the criterion of nodules. In the third pair of images 120 in the bottom row, the object-of-interest 30 has vanished from the VOI on the right. From the surrounding structures we can clearly see the correct correspondence between two VOI's, and we can conclude that the nodule is absent after a period of time.
FIG. 3 is a flowchart of an embodiment of the invention showing how the command flow of the GUI may be achieved, though of course there are many possible variants of command flow in the programming art. Referring to FIG. 3, we see that the image sets are loaded at 200. This step of loading the image sets could alternatively have been executed from box 240 as a file operation. In flow box 210, the GUI awaits a user command, which may be (1) the clicking on an object-of-interest, (2) a user input for change of display, or (3) a menu option. Box 210 would generally be event driven. The remaining boxes show the flow of execution of commands and return to box 210 as has been described above. Upon option 1, flow goes to box 220 where the correlation system is invoked to locate a corresponding VOI in a second image set. A first VOI is also defined about the first object-of-interest. Flow then goes to box 230 where the VOI's and object properties are displayed before control flow returns to the user at box 210. Upon option 2, any one or more of the display changes described above are executed in real time at box 250 before control returns to the user. Upon option 3, any one of standard or specialized menu operations are executed at box 240, including such familiar operations as file operations (e.g., open file, close file, save file, etc.), user and program preferences and settings, access to a help file, or simply exiting the program.
It is therefore shown that this invention would greatly ease and improve the quality of nodule correspondence identification and henceforth the screening and diagnosis of lung cancer. As the result, the clinical throughput and diagnostic accuracy would be greatly improved.
Though the invention has been described with respect to two image sets, the principles of the invention may easily be extended to any greater number of image sets.
The methods of the invention may be implemented as a program of instructions, readable and executable by machine such as a computer, and tangibly embodied and stored upon a machine-readable medium such as a computer memory device.
It is to be understood that all physical quantities disclosed herein, unless explicitly indicated otherwise, are not to be construed as exactly equal to the quantity disclosed, but rather as about equal to the quantity disclosed. Further, the mere absence of a qualifier such as “about” or the like, is not to be construed as an explicit indication that any such disclosed physical quantity is an exact quantity, irrespective of whether such qualifiers are used with respect to any other physical quantities disclosed herein.
While preferred embodiments have been shown and described, various modifications and substitutions may be made thereto without departing from the spirit and scope of the invention. Accordingly, it is to be understood that the present invention has been described by way of illustration only, and such illustrations and embodiments as have been disclosed herein are not to be construed as limiting to the claims.

Claims (27)

1. A graphical user interface, comprising:
a plurality of main displays, each for displaying an image set, said plurality of displays adapted for simultaneously displaying images from different image sets;
a user interactive system for receiving from a user a first location of an object-of-interest in one of said image sets; and
a correlation system for finding and highlighting a second location, corresponding to said first location, of a corresponding second volume-of-interest in at least one of said other image sets.
2. The graphical user interface of claim 1 wherein a first volume-of-interest is defined about said first location of said object of interest.
3. The graphical user interface of claim 2 wherein said first and second volumes-of-interest are rendered as Shaded Surface Displays.
4. The graphical user interface of claim 1 wherein said image sets are rendered as Shaded Surface Displays.
5. The graphical user interface of claim 1 further comprising
displays permitting the user to examine said volumes-of-interest with free viewpoints.
6. The graphical user interface of claim 1 further comprising
one or more data windows for displaying image properties of at least one of said image sets.
7. The graphical user interface of claim 1 further comprising
a lock-scrolling system for synchronized scrolling of two or more image sets.
8. The graphical user interface of claim 1 further comprising
a cartwheel projection system for side-by-side display of two or more cartwheel projection spin windows.
9. The graphical user interface of claim 1 further comprising:
a first property display for displaying physical properties of said object-of-interest;
a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest; and
wherein said first and second property displays may be displayed side-by-side.
10. A program storage device, readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for a graphical user interface for an object-correspondence system, said method steps comprising:
providing a plurality of main displays, each for displaying an image set, wherein said main displays are adapted for the simultaneous display of images from different image sets;
receiving from a user a first location of an object-of-interest in one of said image sets; and
receiving and highlighting a second location of a second volume-of-interest from a correlation system, said second location corresponding to said first location.
11. The storage device of claim 10 wherein a first volume-of-interest is defined about said first location of said object-of-interest.
12. The storage device of claim 11 wherein said first and second volumes-of-interest are rendered as Shaded Surface Displays.
13. The storage device of claim 11 further comprising
machine readable code to allow the user to examine said volumes-of-interest with free viewpoints in 3D in synchronization.
14. The storage device of claim 10 wherein said image sets are rendered as Shaded Surface Displays.
15. The storage device of claim 10 further comprising
one or more data windows for displaying image properties of a volume-of-interest at least one of said image sets.
16. The storage device of claim 10 further comprising
machine-readable code for synchronized scrolling of two or more image sets.
17. The storage device of claim 10 further comprising
machine-readable code for side-by-side display of two or more cartwheel projection spin windows.
18. The storage device of claim 10 further comprising machine readable code for:
providing a first property display for displaying physical properties of said object-of-interest;
providing a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest; and
displaying said first and second property displays side-by-side.
19. A method of interfacing graphically with a user for an object-correspondence identification system, comprising the steps of:
providing a plurality of main displays, each for displaying an image set, wherein said main displays are adapted for the simultaneous display of images from different image sets;
receiving from a user a first location of an object-of-interest in one of said image sets; and
receiving and highlighting a second location of a second volume-of-interest from a correlation system, said second location corresponding to said first location.
20. The method of claim 19 wherein a first volume-of-interest is defined about said first location of said object-of-interest.
21. The method of claim 20 wherein said first and second volumes-of-interest are rendered as Shaded Surface Displays.
22. The method of claim 19 wherein said volumes-of-interest are rendered as Shaded Surface Displays.
23. The method of claim 19 further comprising
providing examination of said volumes-of-interest in 3D with free viewpoints in synchronization.
24. The method of claim 19 further comprising
one or more data windows for displaying image properties of at least one of said image sets.
25. The method of claim 19 further comprising
providing a lock-scrolling system for synchronized scrolling of two or more image sets.
26. The method of claim 19 further comprising
providing a cartwheel projection system for side-by-side display of two or mare cartwheel projection spin windows.
27. The method of claim 19 further comprising:
providing a first property display for displaying physical properties of said object-of-interest;
providing a second property display for displaying physical properties of a second object-of-interest located within said second volume-of-interest; and
displaying said first and second property displays side-by-side.
US10/090,314 2002-03-04 2002-03-04 System GUI for identification and synchronized display of object-correspondence in CT volume image sets Expired - Fee Related US7296239B2 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US10/090,314 US7296239B2 (en) 2002-03-04 2002-03-04 System GUI for identification and synchronized display of object-correspondence in CT volume image sets
DE10392341T DE10392341T5 (en) 2002-03-04 2003-03-04 System GUI for identification and synchronized display of object correspondence in CT volume image sets
JP2003575341A JP2005518916A (en) 2002-03-04 2003-03-04 System GUI for object correspondence identification and synchronized display in CT volume image sets
PCT/US2003/006395 WO2003077203A2 (en) 2002-03-04 2003-03-04 Graphical user interface for object-correspondence in ct volume image sets
CNA038052431A CN1639739A (en) 2002-03-04 2003-03-04 A graphical user interface of object consistency in CT volume image sets

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US10/090,314 US7296239B2 (en) 2002-03-04 2002-03-04 System GUI for identification and synchronized display of object-correspondence in CT volume image sets

Publications (2)

Publication Number Publication Date
US20030164860A1 US20030164860A1 (en) 2003-09-04
US7296239B2 true US7296239B2 (en) 2007-11-13

Family

ID=27803999

Family Applications (1)

Application Number Title Priority Date Filing Date
US10/090,314 Expired - Fee Related US7296239B2 (en) 2002-03-04 2002-03-04 System GUI for identification and synchronized display of object-correspondence in CT volume image sets

Country Status (5)

Country Link
US (1) US7296239B2 (en)
JP (1) JP2005518916A (en)
CN (1) CN1639739A (en)
DE (1) DE10392341T5 (en)
WO (1) WO2003077203A2 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060050943A1 (en) * 2002-12-03 2006-03-09 Masahiro Ozaki Computer-aided diagnostic apparatus
US20060174212A1 (en) * 2005-02-01 2006-08-03 Eastman Kodak Company Information presentation on wide-screen displays
US20070064982A1 (en) * 2005-09-19 2007-03-22 General Electric Company Clinical review and analysis work flow for lung nodule assessment
US20070230758A1 (en) * 2006-03-31 2007-10-04 Siemens Medical Solutions Usa, Inc. Cross reference measurement for diagnostic medical imaging
EP2109080A1 (en) 2008-04-09 2009-10-14 IBBT vzw A method and device for processing and presenting medical images
US20100053211A1 (en) * 2008-06-27 2010-03-04 Vala Sciences, Inc. User interface method and system with image viewer for management and control of automated image processing in high content screening or high throughput screening
US20100192084A1 (en) * 2009-01-06 2010-07-29 Vala Sciences, Inc. Automated image analysis with gui management and control of a pipeline workflow
US20100293505A1 (en) * 2006-08-11 2010-11-18 Koninklijke Philips Electronics N.V. Anatomy-related image-context-dependent applications for efficient diagnosis
US9064448B1 (en) * 2011-08-31 2015-06-23 Google Inc. Digital image comparison

Families Citing this family (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7127684B2 (en) * 2002-11-26 2006-10-24 Ge Informational Systems Technologies, Inc. Synchronized magnification system and method for images
US20040175764A1 (en) * 2003-01-06 2004-09-09 Hiroto Nishiyama Image processing apparatus, image processing program, recording medium, and image processing method
JP4401140B2 (en) * 2003-10-24 2010-01-20 株式会社日立メディコ Diagnostic imaging support device
US7301535B2 (en) 2004-09-02 2007-11-27 Siemens Medical Solutions Usa, Inc. 3D summary display for reporting of organ tumors
GB2418094B (en) 2004-09-10 2010-05-12 Medicsight Plc User interface for CT scan analysis
US7787672B2 (en) 2004-11-04 2010-08-31 Dr Systems, Inc. Systems and methods for matching, naming, and displaying medical images
US7970625B2 (en) 2004-11-04 2011-06-28 Dr Systems, Inc. Systems and methods for retrieval of medical data
US7885440B2 (en) 2004-11-04 2011-02-08 Dr Systems, Inc. Systems and methods for interleaving series of medical images
US7660488B2 (en) 2004-11-04 2010-02-09 Dr Systems, Inc. Systems and methods for viewing medical images
US7920152B2 (en) 2004-11-04 2011-04-05 Dr Systems, Inc. Systems and methods for viewing medical 3D imaging volumes
US20060135865A1 (en) * 2004-11-23 2006-06-22 General Electric Company Method and apparatus for synching of images using regions of interest mapped by a user
US20060277488A1 (en) * 2005-06-07 2006-12-07 Eastman Kodak Company Information presentation on wide-screen displays
JP4794993B2 (en) * 2005-11-17 2011-10-19 富士フイルムRiファーマ株式会社 Image diagnosis support apparatus and image display method
WO2008018029A1 (en) * 2006-08-11 2008-02-14 Koninklijke Philips Electronics N.V., Selection of datasets from 3d renderings for viewing
US8223143B2 (en) 2006-10-27 2012-07-17 Carl Zeiss Meditec, Inc. User interface for efficiently displaying relevant OCT imaging data
US7953614B1 (en) 2006-11-22 2011-05-31 Dr Systems, Inc. Smart placement rules
US8816959B2 (en) 2007-04-03 2014-08-26 General Electric Company Method and apparatus for obtaining and/or analyzing anatomical images
US8290303B2 (en) 2007-10-11 2012-10-16 General Electric Company Enhanced system and method for volume based registration
DE102008025535B4 (en) * 2008-05-28 2014-11-20 Siemens Aktiengesellschaft Method for viewing tubular anatomical structures, in particular vascular structures, in medical 3D image recordings
US8380533B2 (en) 2008-11-19 2013-02-19 DR Systems Inc. System and method of providing dynamic and customizable medical examination forms
US8712120B1 (en) 2009-09-28 2014-04-29 Dr Systems, Inc. Rules-based approach to transferring and/or viewing medical images
WO2011066222A1 (en) * 2009-11-25 2011-06-03 Vital Images, Inc. User interface for providing clinical applications and associated data sets based on image data
US9996971B2 (en) 2009-11-25 2018-06-12 Carestream Health, Inc. System providing companion images
CN102117168A (en) * 2009-12-31 2011-07-06 英华达(上海)电子有限公司 Portable electronic book reading device and method thereof for data processing
US9092727B1 (en) 2011-08-11 2015-07-28 D.R. Systems, Inc. Exam type mapping
US8944597B2 (en) 2012-01-19 2015-02-03 Carl Zeiss Meditec, Inc. Standardized display of optical coherence tomography imaging data
US9495604B1 (en) 2013-01-09 2016-11-15 D.R. Systems, Inc. Intelligent management of computerized advanced processing
US9420945B2 (en) 2013-03-14 2016-08-23 Carl Zeiss Meditec, Inc. User interface for acquisition, display and analysis of ophthalmic diagnostic data
CN104077177A (en) * 2013-03-28 2014-10-01 上海西门子医疗器械有限公司 Method for processing reconstruction task and computer tomography equipment
CN103412762B (en) * 2013-08-29 2017-04-05 宇龙计算机通信科技(深圳)有限公司 Terminal and terminal operation method
JP6344039B2 (en) * 2014-04-28 2018-06-20 富士通株式会社 Image display device, image display method, and program
US20170046483A1 (en) 2015-04-30 2017-02-16 D.R. Systems, Inc. Database systems and interactive user interfaces for dynamic interaction with, and comparison of, digital medical image data
JP6598565B2 (en) * 2015-08-06 2019-10-30 キヤノン株式会社 Image processing apparatus, image processing method, and program
CN111160812B (en) * 2020-02-17 2023-08-29 杭州依图医疗技术有限公司 Diagnostic information evaluation method, display method, and storage medium
CN111261284A (en) * 2020-02-05 2020-06-09 杭州依图医疗技术有限公司 Medical image-based diagnostic information processing method and device and storage medium
CN111261285A (en) * 2020-02-07 2020-06-09 杭州依图医疗技术有限公司 Display method, interaction method and storage medium of diagnostic information interface
CN112925461A (en) * 2021-02-23 2021-06-08 上海商汤智能科技有限公司 Image processing method and device, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6058322A (en) 1997-07-25 2000-05-02 Arch Development Corporation Methods for improving the accuracy in differential diagnosis on radiologic examinations
US6597173B1 (en) * 2002-07-10 2003-07-22 Ge Medical Systems Global Technology Co., Llc Method and apparatus for reconstructing zoom MR images
US6785410B2 (en) * 1999-08-09 2004-08-31 Wake Forest University Health Sciences Image reporting method and system
US6798412B2 (en) * 2000-09-06 2004-09-28 Idelix Software Inc. Occlusion reducing transformations for three-dimensional detail-in-context viewing

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6058322A (en) 1997-07-25 2000-05-02 Arch Development Corporation Methods for improving the accuracy in differential diagnosis on radiologic examinations
US6785410B2 (en) * 1999-08-09 2004-08-31 Wake Forest University Health Sciences Image reporting method and system
US6798412B2 (en) * 2000-09-06 2004-09-28 Idelix Software Inc. Occlusion reducing transformations for three-dimensional detail-in-context viewing
US6597173B1 (en) * 2002-07-10 2003-07-22 Ge Medical Systems Global Technology Co., Llc Method and apparatus for reconstructing zoom MR images

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Fan Li et al., "Automatic Segmentation of Pulmonary Nodules by Using Dynamic 3D Cross-correlation for Interactive CAD Systems," Medical Imaging 2002, XP008022811.
International Search Report, Apr. 3, 2003.
Ko Jane P. et al., Chest CT : Automated Nodule Detection and Assessment of Change over Time-Preliminary Experience, Radiology, 2001 218:267-273 XP-002266047.

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060050943A1 (en) * 2002-12-03 2006-03-09 Masahiro Ozaki Computer-aided diagnostic apparatus
US9168007B2 (en) * 2002-12-03 2015-10-27 Kabushiki Kaisha Toshiba Computer-aided diagnostic apparatus
US20060174212A1 (en) * 2005-02-01 2006-08-03 Eastman Kodak Company Information presentation on wide-screen displays
US20070064982A1 (en) * 2005-09-19 2007-03-22 General Electric Company Clinical review and analysis work flow for lung nodule assessment
US8732601B2 (en) 2005-09-19 2014-05-20 General Electric Company Clinical review and analysis work flow for lung nodule assessment
US7817835B2 (en) * 2006-03-31 2010-10-19 Siemens Medical Solutions Usa, Inc. Cross reference measurement for diagnostic medical imaging
US20070230758A1 (en) * 2006-03-31 2007-10-04 Siemens Medical Solutions Usa, Inc. Cross reference measurement for diagnostic medical imaging
US20100293505A1 (en) * 2006-08-11 2010-11-18 Koninklijke Philips Electronics N.V. Anatomy-related image-context-dependent applications for efficient diagnosis
EP2109080A1 (en) 2008-04-09 2009-10-14 IBBT vzw A method and device for processing and presenting medical images
US20090257657A1 (en) * 2008-04-09 2009-10-15 Temmermans Frederik Method and device for processing and presenting medical images
US20100053211A1 (en) * 2008-06-27 2010-03-04 Vala Sciences, Inc. User interface method and system with image viewer for management and control of automated image processing in high content screening or high throughput screening
US20100192084A1 (en) * 2009-01-06 2010-07-29 Vala Sciences, Inc. Automated image analysis with gui management and control of a pipeline workflow
US8861810B2 (en) 2009-01-06 2014-10-14 Vala Sciences, Inc. Automated image analysis with GUI management and control of a pipeline workflow
US9064448B1 (en) * 2011-08-31 2015-06-23 Google Inc. Digital image comparison
US9449582B2 (en) 2011-08-31 2016-09-20 Google Inc. Digital image comparison
US10199013B2 (en) 2011-08-31 2019-02-05 Google Llc Digital image comparison

Also Published As

Publication number Publication date
DE10392341T5 (en) 2005-03-03
JP2005518916A (en) 2005-06-30
US20030164860A1 (en) 2003-09-04
CN1639739A (en) 2005-07-13
WO2003077203A3 (en) 2004-04-01
WO2003077203A2 (en) 2003-09-18

Similar Documents

Publication Publication Date Title
US7296239B2 (en) System GUI for identification and synchronized display of object-correspondence in CT volume image sets
EP2048621B1 (en) Method and apparatus for volume rendering of medical data sets
US6944330B2 (en) Interactive computer-aided diagnosis method and system for assisting diagnosis of lung nodules in digital volumetric medical images
US7133546B2 (en) Digital medical image analysis
EP3267894B1 (en) Retrieval of corresponding structures in pairs of medical images
US6901277B2 (en) Methods for generating a lung report
EP2486548B1 (en) Interactive selection of a volume of interest in an image
US8442280B2 (en) Method and system for intelligent qualitative and quantitative analysis of digital radiography softcopy reading
US20070276214A1 (en) Systems and Methods for Automated Segmentation, Visualization and Analysis of Medical Images
US8150120B2 (en) Method for determining a bounding surface for segmentation of an anatomical object of interest
US20090063118A1 (en) Systems and methods for interactive navigation and visualization of medical images
US20070019849A1 (en) Systems and graphical user interface for analyzing body images
US20010055016A1 (en) System and method for volume rendering-based segmentation
WO2013028762A1 (en) Method and system for integrated radiological and pathological information for diagnosis, therapy selection, and monitoring
EP2116974B1 (en) Statistics collection for lesion segmentation
US20080089584A1 (en) Viewing glass display for multi-component images
US9361711B2 (en) Lesion-type specific reconstruction and display of digital breast tomosynthesis volumes
US20160171158A1 (en) Medical imaging apparatus and method using comparison image
JP2007512064A (en) Method for navigation in 3D image data
JPH11306264A (en) Computer aided diagnostic device
US20230419602A1 (en) Rendering and displaying a 3d representation of an anatomical structure
JP2008029703A (en) Three-dimensional image display device
WO2016181037A1 (en) Computer aided medical imaging report
Maxime et al. Fast interactive volume rendering method for adjustable vessel segmentation visualization

Legal Events

Date Code Title Description
AS Assignment

Owner name: SIEMENS CORPORATE RESEARCH, INC., NEW JERSEY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHEN, HONG;REEL/FRAME:013020/0338

Effective date: 20020607

AS Assignment

Owner name: SIEMENS CORPORATE RESEARCH, INC., NEW JERSEY

Free format text: RE-RECORD TO ADD ASSIGNEE'S NAME PREVIOUSLY RECORDED AT REEL/FRAME 013020/0338;ASSIGNORS:SHEN, HONG;QIAN, JIANZHONG;FAN, LI;REEL/FRAME:013776/0876

Effective date: 20020607

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: SIEMENS CORPORATION,NEW JERSEY

Free format text: MERGER;ASSIGNOR:SIEMENS CORPORATE RESEARCH, INC.;REEL/FRAME:024185/0042

Effective date: 20090902

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: SIEMENS AKTIENGESELLSCHAFT, GERMANY

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SIEMENS CORPORATION;REEL/FRAME:028452/0780

Effective date: 20120627

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20151113