DE102011084348A1 - Miniaturized opto-electronic system for spectral analysis - Google Patents

Abstract

The invention relates to a miniaturized optoelectronic system for generating static or moving images of scenes or individual objects (11) and for determining and evaluating the spectral properties of the objects (11) within a scene or also of individually imaged objects (11). According to the invention, a system of this kind comprises optical components and beam paths for generating static or moving images of a scene or a single object (11), optical components and beam paths for determining the spectral properties of one or more objects (11) contained in the scene individual object (11), - at least one image sensor as an optoelectronic converter, - electronic components for processing the output signals of the image sensor, - an information output unit, designed to display the results sensually, preferably visually perceptible representation of the spectral properties in association with the objects (11), and - Means for powering electronic components, - wherein an embodiment of the system in the form of a hand-held device, also referred to as a handheld, is provided.

Description

The invention relates to a miniaturized optoelectronic system for generating static or moving images of scenes or individual objects and for determining and evaluating the spectral properties of the objects within a scene or of individually imaged objects.

A study is known about commercial devices in the form of mobile phones, which are equipped with microscopic imaging properties as well as with the ability to spectral analysis of an object to be examined. This study is published under "Cell Phone Based Platform for Biomedical Device Development and Education Applications; Zachary J. Smith at all; Center for Biophotonics Science and Technology, University of California Davis, Sacramento, California, USA ,

Here, a mobile phone was coupled with a single-channel spectrometer, which has a bandwidth of 300 nm and a spectral resolution of less than 5 nm. The determination of white light transmission spectra and fluorescence spectra is thus possible. The use of this device combination, referred to as a mobile phone spectrometer, is provided in connection with medically relevant applications. The device base used was a GSM mobile phone suitable for media playback, which has a video camera, mobile Internet access via e-mail and GPS cards and is currently known under the trade name iPhone.

Furthermore, in US 2010/0309454 A1 describes a compact spectrometer coupled to a mobile phone or integrated in a miniaturized version in a mobile phone. Impliziert are also the power supply, data storage, signal processing and real-time display of the results. The results can be communicated immediately via wireless communication to a remote station or to other mobile phones or their users. For example, for diabetics, it is possible to perform blood glucose measurements in a non-invasive manner at home and to transmit the results to the doctor.

A disadvantage of the aforementioned arrangements is that a comprehensive and precise determination of the spectral properties of imaged scenes or objects in real time is not possible and the output of inferential information, for example in the sense of recommendations for action for the user, which relate to the determination and evaluation results , not intended or - measured against existing needs - only limited and therefore insufficiently possible.

The invention described below is based on the object to remedy the deficiencies inherent in the prior art.

• – optische Bauelemente und Strahlengänge zur Erzeugung statischer oder bewegter Bilder einer Szene oder eines einzelnen Objektes,
• – optische Bauelemente und Strahlengänge zur Bestimmung der Spektraleigenschaften eines oder mehrerer in der Szene enthaltener Objekte oder eines einzelnen Objektes,
• – mindestens einen Bildsensor als optoelektronischen Wandler,
• – elektronische Bauelemente, ausgebildet zur Verarbeitung der Ausgangssignale des Bildsensors,
• – eine Informationsausgabeeinheit, ausgebildet zur Ergebnisdarstellung sinnlich, vorzugsweise visuell wahrnehmbaren Darstellung der Spektraleigenschaften in Zuordnung zu den Objekten, und
• – Mittel zur Stromversorgung der elektronischen Bauelemente,
• – wobei eine Ausführung des Systems in Form eines Handgerätes, auch als Handheld bezeichnet, vorgesehen ist.

According to the invention, a miniaturized optoelectronic system of the type mentioned in the introduction comprises

• Optical components and beam paths for generating static or moving images of a scene or a single object,
• Optical components and optical paths for determining the spectral properties of one or more objects contained in the scene or a single object,
• At least one image sensor as optoelectronic transducer,
• Electronic components designed to process the output signals of the image sensor,
• An information output unit designed to display the results in a sensually, preferably visually perceptible representation of the spectral properties in association with the objects, and
• - means for powering the electronic components,
• - Wherein an embodiment of the system in the form of a hand-held device, also referred to as a handheld, is provided.

A handheld or handheld device according to the present invention is a portable electronic device with its own power supply for different applications. It is so small and light that it can be held in use in one hand only, hence the term Handheld, held by English in the hand (cited from Wikipedia, 31.08.2011).

As beam paths in the sense of the present invention, the paths of the light through the optical part of the system are to be understood. In different embodiments of the system according to the invention, the beam paths for image generation and the beam paths for determining the spectral properties run at least in sections separately or together, and the image generation and spectral analysis are provided simultaneously or sequentially.

In a preferred embodiment, the optical means for determining the spectral properties are designed as multichannel spectrometers and provided to determine spectrums for several wavelengths of the illumination and / or measurement light independently, preferably simultaneously with the generation of images of scenes or objects whose spectral Properties are to be determined.

Included in the inventive concept is the training in the form of a hyperspectral multi-channel spectrometer, which is suitable for spectral analysis in the wavelength range from ultraviolet to long-wave infrared and is provided in each of the channels with up to 250 different, each one wavelength or wavelength range corresponding support points.

• – ein polychromatischer Silizium basierter Bildsensor,
• – ein polychromatischer Indium-Gallium-Arsenit (InGaAs) basierter Bildsensor,
• – mehrere monochromatische Einzelsensoren auf Silizium- oder Indium-Gallium-Arsenid-Basis, die zusammenwirkend einen bestimmten, spektral zu analysierenden Wellenlängenbereich überdecken, oder
• – Kombinationen aus polychromatischen und monochromatischen Zeilen- oder Flächensensoren auf Silizium- und / oder Indium-Gallium-Arsenid-Basis, die ebenfalls einen bestimmten, spektral zu analysierenden Wellenlängenbereich überdecken.

Optionally, it may be provided as an optoelectronic converter

• A polychromatic silicon based image sensor,
• A polychromatic indium gallium arsenite (InGaAs) based image sensor,
• A plurality of monochromatic silicon or indium gallium arsenide based single sensors cooperatively covering a particular wavelength range to be spectrally analyzed, or
• - combinations of polychromatic and monochromatic line or surface sensors on silicon and / or indium-gallium-arsenide basis, which also cover a specific spectrally analyzed wavelength range.

The system according to the invention is embodied in special embodiments for producing two- or three-dimensional images of scenes or objects, and the spectral analysis is carried out for one or more individual lines, for an area or for a volume of an object selected from a scene or individually recorded.

The generation of three-dimensional images is preferably carried out using the known TOF = Time Of Flight principle. This is particularly advantageous for applications that require a time-resolved spectral analysis. In this case, a photonic mixer, also referred to as a PMD sensor, is advantageously used as the image sensor.

The means for imaging and the means for determining the spectral properties are either housed within a common compact housing or designed as separate modules. For both options embodiments are provided, which allow the one-handed holding and operating due to their size and weight.

In the case of execution in separate modules they are coupled to each other via interfaces. Advantageously, a dispersive optics, for example in the form of an optical grating, prism or filter, are arranged upstream of the optical elements for image generation, in particular an objective belonging to the imaging beam path.

In order to achieve a compensation of the image blur, which arises due to the dispersion in the direction of the Z-axis of the Rowlandkreis accordingly, as well as to compensate for further imaging blurring in planes which are inclined to the plane defined by the Z-axis and the dispersion direction plane Image stack taken from different focal planes, the individual images of these stacks with respect to the image sharpness of preferably relevant pixels evaluated and obtained from the results images of the scene or objects whose image sharpness is corrected.

The focus variation is made, for example, by stepwise changing the distance between the lens and the focal plane of the dispersion element. With appropriate programming, an existing autofocusing device can be temporarily used for this purpose.

Alternatively, the use of DOEs or other optically correcting elements is conceivable in order to compensate the image blur.

For illuminating the scene or the object to be recorded and / or for spectral analysis, the respective natural ambient light or the light emitted by an artificial source is provided. The coupling of the measuring light coming from the scene or the object into the beam paths for the spectral analysis takes place, for example, by means of free-ray optics or optical waveguides. Within the scope of the invention is also the separate or integrated equipment of the opto-electronic system with an illumination light source that emits fluorescence exciting light.

If artificial light sources are used, they are preferably connected to means for the defined influencing of the intensity, the wavelength and / or the polarization of the emitted light. The use of polarization elements enables the processing of phase information, which is particularly advantageous in spectral ellipsometry for the purpose of investigating layer thicknesses and optical properties of different materials. In this case, polarized light is irradiated onto the sample, and based on the amplitude and phase information or the ellipsometric parameters Y and D of the reflected beam is concluded that the polarizing properties of the sample.

In a further embodiment of the invention, the generation of instructions for action for the user is provided, which are generated in the form of conclusions from the results of the spectral analysis. Included in the inventive concept is the additional Linking the results of the spectral analysis with other information relating to scenes and objects, such as metadata. These are either stored internally in the system, stored in an external data store assigned to the system, or made publicly accessible by communication and retrievable with the system.

With regard to the latter, the optoelectronic system according to the invention is equipped, for example, with means for wireless integration into a communications network, in particular for image and sound transmission, for transmission of measurement results to external data memories and / or for retrieving information from external data memories. Such information is for example reference values for the calibration of the means for image generation, but preferably for the calibration of the means for spectral analysis. In particular, the calibration of the intensity and / or the wavelength of the illumination light can be provided.

In a further embodiment, which is likewise within the scope of the invention, the optoelectronic system according to the invention is equipped with means for guiding the user in the search for such objects within arbitrary scenes which have certain spectral properties.

This route guidance is achieved by determining, for example by means of GPS and / or compass, the spatial orientation of the imaged scene or of the imaged object and correlating it with the temporal gradient of a change arbitrarily or deliberately caused by the user to the measured absolute or relative spatial spectral information becomes. The return information thus obtained to the user allows the route guidance of the system to areas that are relevant to the user in connection with the respective application.

Further information supporting the context of the measurement, e.g. Temperature, are taken into account in the calculation of the gradient space information in addition.

The real-time feedback of the measurement information is e.g. in the display by means of supporting points offering grids, arrows, Falschfarbenflächen, or by means of haptic instant information to the user, such as acoustically and / or in the form of vibrations.

The feedback to the user does not necessarily have to be in the same place, that is to say the user can obtain the information spatially separated via the communication interfaces of the system according to the invention, such as e.g. TCP / IP, Bluetooth etc.

In connection with the route guidance, for example, the linking of objects that have been determined to be interesting on account of their spectral properties, with image-covered terrain forms, is also provided for certain applications.

The information output unit has, for example, an LED or OLED display for visually perceptible presentation of results.

In a further advantageous embodiment, the means for image generation are designed as a light field camera system. Light field camera systems are known as plenoptic cameras. They capture the 4D light field of a scene, which not only the position and intensity of a light beam incident on the image sensor is known, but also its direction of incidence. For light field measurement, for example, a grid of several microlenses is arranged in front of the image sensor. Thus, even without autofocusing, very large depth of field shots are advantageously achieved with the least possible image blur.

The imaging means, preferably embodied as a camera, advantageously have a sensor matrix, for example embodied in the form of a CCD image sensor, which is connected to a drive for temporarily combining adjacent pixels into pixel blocks in order to achieve a higher photosensitivity and at the same time a higher read-out speed. This control can be made spatially and / or temporally staggered with respect to the pixels of the image sensor.

Within the scope of the invention, the equipment of the optoelectronic system with acceleration sensor, gyroscope and inclinometer as well as means for integration into the Global Positioning System (GPS) are optional as well. By means of the inclinometer and GPS, for example, the determination of the angle of incidence of the measuring and / or illumination light on an object to be imaged is provided in relation to the optical axis of the imaging optics.

In the context of the present invention, the term illumination light should be understood to mean the light directed onto the scene or the object, the term measuring light being understood to mean the light to be evaluated by the scene or the object and corresponding to the respective application.

If artificial sources are provided for the illumination or measuring light, these can optionally be operated continuously or pulsed. If the system according to the invention is used as a spectral video Operated system, in the latter case, the sequence in the recording of the moving images with the pulse frequency of the light source to be synchronized so that the recording always takes place upon illumination of the scene or the object, so that periodically repeating events or periodically recurring events made visible to be able to determine, for example, temporally progressive changes.

In a further advantageous embodiment, means are provided for classifying imaged objects on the basis of their spectral properties, the classification being carried out for example under chemically, medically or agronomically revealing aspects.

• – eine Überlagerung des Livebildes mit einer spektralen Messung vorgenommen werden, bei der die Sequenz der Bilderfassung größer ist als die Sequenz, mit der die Spektralanalyse erfolgt. Der Bildsensor wird dabei bereichsweise für die spektrale Messung und bereichsweise für das Livebild genutzt. Diesbezüglich sind vorzugsweise zwei Kameras im System vorgesehen, von denen eine der spektralen Messung und die zweite dem Livebild vorbehalten ist. Im Livebild wird – für den Nutzer erkennbar – eine Markierung des Bildanteiles vorgenommen, in dem die spektrale Messung erfolgt,
• – eine Überlagerung des Livebildes mit einer spektralen Messung vorgenommen werden, bei der die Sequenz der Bilderfassung der Sequenz entspricht, mit der die Spektralanalyse erfolgt. Hierbei werden die jeweils in Echtzeit aufgenommenen Bild- und Spektralinformationen in gleichen Bildsequenzen verarbeitet und aufbereitet. Dies bietet den Vorteil, dass zeitkritische bzw. sich individuell verändernde Prozesse genauestens analysiert und die Prozessabläufe zugleich an die physiologischen Gegebenheiten des Menschen angepasst werden können, der eine Bildwiederholfrequenz von 16 Hz wahrnehmen kann,
• – eine Überlagerung des Livebildes mit einer spektralen Messung vorgenommen werden, bei der die Sequenz der Bilderfassung kleiner ist als die Sequenz, mit der die Spektralanalyse erfolgt. Dies ist dann vorteilhaft, wenn das Messobjekt spektral inhomogen ist. Eine große Anzahl von Einzelmessungen während einer kleineren Anzahl von Bilderzeugungen verbessert hierbei die Genauigkeit der Gesamtmessung.

The miniaturized optoelectronic system according to the invention offers significant advantages in contrast to the systems available according to the prior art. Thus, according to the invention

• - Overlay the live image are made with a spectral measurement in which the sequence of image acquisition is greater than the sequence with which the spectral analysis is performed. The image sensor is used in some areas for the spectral measurement and partially for the live image. In this regard, two cameras are preferably provided in the system, of which one of the spectral measurement and the second is reserved for the live image. In the live image - recognizable for the user - a marking of the image portion is made in which the spectral measurement takes place,
• - A superimposition of the live image are made with a spectral measurement in which the sequence of image acquisition corresponds to the sequence with which the spectral analysis is performed. Here, the image and spectral information recorded in real time are processed and processed in the same image sequences. This offers the advantage that time-critical or individually changing processes can be analyzed in detail and the processes can be adapted to the physiological conditions of humans at the same time, which can perceive a refresh rate of 16 Hz,
• - Overlay the live image are made with a spectral measurement in which the sequence of image acquisition is smaller than the sequence with which the spectral analysis is performed. This is advantageous if the measurement object is spectrally inhomogeneous. A large number of individual measurements during a smaller number of image generations improves the accuracy of the overall measurement.

The system according to the invention can be operated as a spectral video stroboscope system, wherein a synchronization to the operating frequency of the existing light sources for the purpose of time allocation of lighting and recording is provided. Alternatively, applications using the time sequence are useful, in which the dynamic history is to be visualized.

For applications in connection with time-resolved spectroscopy, the system according to the invention is additionally equipped with a shutter, which preferably has a reaction time in the nanosecond range. The shutter serves to link the beam path for image generation with the beam path for spectral measurement and makes it possible to perform spectral measurement and image generation sequentially in time with the shutter control, which is advantageous in terms of time-resolved spectroscopy, especially with rapid temporal changes to the measurement objects.

If the optoelectronic system according to the invention is equipped with a 3D camera, which is preferably provided with the time-of-flight function, the spectral measurements can be carried out with time resolution. If a spectroscopic intent is used for the spectral measurement, as explained in greater detail below on exemplary embodiments, the distance determination to the object is nevertheless possible. Furthermore, it is hereby possible to measure time differences for the measurement of information from the temporal signal behavior of certain substances and applications and to provide them for further processing, as for example in the so-called FLIM technique (Fluorescence Lifetime Imaging Microscopy), which uses ion-sensitive fluorescent dyes for measuring intracellular ion concentrations , Here, the fact is used that changes the fluorescence lifetime, that is, the average residence time of electrons in the excited state, with the ion concentration.

As already partially mentioned above, ambient light or artificial light can be used to illuminate the scenes or objects to be recorded. As a source of artificial light, internal light sources are considered, optionally with an optical attachment, or external light sources, which can also be operated pulsed, for example, as a flash for stroboscopic lighting or in connection with the 3D equipment based on TOF.

• – Verwendung des Displays der Informationsausgabeeinheit als Beleuchtung,
• – Verwendung einer oder mehrerer monochromatischer Lichtquellen, die als Stützstellen für die Spektralmessungen dienen, zum parallelen oder seriellen Beleuchten, optional auch im Takt der Aufnahme bewegter Bilder. Dabei kann die spektrale Auflösung des Gesamtsystems auch realisiert werden, indem die Anzahl der Lichtquellen den gesamten spektralen Wellenlängenbereich – auch außerhalb des RGB-Farbraumes – überdeckt,
• – Verwendung einer Lichtquelle zur Fluoreszenzanregung bei flächiger Beleuchtung,
• – Verwendung von natürlichen Lichtquellen zur fluoreszenzspektroskopischen Anregung,
• – zeitlich aufeinander folgende Beleuchtung der Szene bzw. des Objektes mit Licht unterschiedlicher optischer Eigenschaften, wie Polarisation, Intensität, Wellenlänge oder Spektralbereich,
• – Verwendung des natürlichen Umgebungslichtes unter Zuhilfenahme eines oder mehrerer polarisierender Elemente zur Beleuchtung,
• – Modulation der natürlichen, internen oder externen Beleuchtung, z.B. im Zusammenhang mit zeitaufgelösten Applikationen,
• – Steuerung und / oder Regelung der spektralen und zeitlichen Intensität des Beleuchtungslichtes,
• – diffuse Beleuchtung, beispielsweise mittels zwischengeschaltetem Mikrolinsenarray oder eingefügter Planspiegeloptik, um Energie- bzw. Lichtverluste zu vermeiden,
• – individuelle Programmierung der vorhandenen Lichtquellen bezüglich der Intensität, Wellenlänge und / oder Polarisation des abgestrahlten Lichtes,
• – Verwendung adaptiver optischer Elemente in den Strahlengängen des Beleuchtungs- und/oder Messlichtes, um die Wellenfront zu beeinflussen und dadurch beispielsweise Abbildungsfehler der beteiligten optischen Baugruppen zu korrigieren.

The following variants are within the scope of the invention:

• Use of the display of the information output unit as lighting,
• - Using one or more monochromatic light sources, which serve as support points for the spectral measurements, for parallel or serial illumination, optionally also in time with the Take moving pictures. In this case, the spectral resolution of the overall system can also be realized by the number of light sources covers the entire spectral wavelength range - even outside the RGB color space -
• Use of a light source for fluorescence excitation in planar illumination,
• Use of natural light sources for fluorescence spectroscopic excitation,
• Temporally successive illumination of the scene or of the object with light of different optical properties, such as polarization, intensity, wavelength or spectral range,
• Use of the natural ambient light with the aid of one or more polarizing elements for illumination,
• Modulation of natural, internal or external illumination, eg in connection with time-resolved applications,
• Control and / or regulation of the spectral and temporal intensity of the illumination light,
• Diffuse illumination, for example by means of an interposed microlens array or inserted plane mirror optics, in order to avoid energy or light losses,
• Individual programming of the existing light sources with regard to the intensity, wavelength and / or polarization of the emitted light,
• - Use of adaptive optical elements in the beam paths of the illumination and / or measurement light to influence the wavefront and thereby correct, for example, aberrations of the participating optical assemblies.

In order to calibrate the intensity of the illumination light, it is provided, for example, to decouple a part of the light coming from the light source and to couple it as light with a reference spectrum directly into the spectroscopic beam path. This type of referencing can preferably be done in parallel, but also serially.

In the case of the use of a multichannel spectrometer, the reference is "white" on one channel during parallel referencing, and the measurement information on the other channels. From an area of the image sensor, on which no light falls, one receives the reference "black". Depending on the applicative task to be solved, however, the measurement of an additional dark signal can also be provided. The referencing preferably takes place simultaneously and with each image.

In the case of using a single-channel spectrometer, these operations are performed serially. In this regard, for example, in DE 195 28 855 A1 describes a device in which renewed referencing between the measurements are possible with little effort. For this purpose, a separate reference beam path is used, and the measurement beam path is merged via a Y light guide with the reference beam path, wherein in each light guide branch a switchable shutter is arranged. The Y-light guide with shutter serves as an optical switch, and the cross section of the common beam path between the switch and the spectrometer is split between the reference beam and the beam path. This common beam path is coupled into the inlet opening of the spectrometer.

A second camera, for example with an installation position with respect to the detection or light entry direction offset by an angle of 180 ° to the first camera, can be used for reference measurement for the intensity of the ambient light, regardless of which light sources for object or Scene lighting are provided. Alternatively, however, a use of the optoelectronic system according to the invention is also possible without a white reference, which - although not detrimental for certain applications - reduces the accuracy of measurement, but at the same time advantageously reduces the production costs.

In order to calibrate the wavelength of the illumination light, it is provided, for example, to subject the means for determining the spectral properties to a one-time calibration immediately after the system has been manufactured. If necessary, or each time a predetermined period has elapsed, then e.g. be recalibrated with designed as line sources light sources. For this purpose, a selection is made manually or automatically from a data memory in which the parameters of a plurality of line sources are stored.

The transmission of information, such as spectral values, image data, video sequences u.a. takes place between the individual functional groups or modules via interfaces within the system or bidirectionally by wired or wireless communication with external units, including subordinate evaluation routines and taking into account the application-related use.

• – spektrale Informationen mit beispielsweise chemometrisch relevanten Aussagen werden mit anderen Informationen gekoppelt und als „augmented reality“ dargestellt, worunter im Sinne der vorliegenden Erfindung die computergestützte Erweiterung der Realitätswahrnehmung zu verstehen ist, die alle menschlichen Sinneswahrnehmungen einbezieht, oder
• – die Informationen können für maschinelles Sehen („maschine vision“) genutzt werden, wobei im Rahmen der Erfindung alle Formen der computergestützten Lösung von Aufgabenstellungen in Frage kommen, die sich an den Fähigkeiten des menschlichen visuellen Systems orientieren, von der Kommunikation bis zu haptischer Wahrnehmung z.B. für Blinde. Die Informationsausgabe an den Nutzer erfolgt dabei applikationsbezogen vorteilhaft so, dass unerwünschte Störungen der aufzunehmenden Szene bzw. des aufzunehmenden Objektes, etwa durch akustische Signale, vermieden werden.

The feedback to the user is realized by the following procedures:

• - Spectral information with, for example, chemometrically relevant statements are coupled with other information and represented as "augmented reality", which in the context of the present invention, the computer-aided extension of the perception of reality is to be understood, which includes all human sensory perceptions, or
• The information can be used for machine vision, in which All forms of computer-aided solution of tasks in question, which are based on the capabilities of the human visual system, from communication to haptic perception eg for the blind. The information output to the user takes place in an application-related advantageous manner so that unwanted disturbances of the scene to be recorded or of the object to be recorded, for example by acoustic signals, are avoided.

The communication between users of the optoelectronic systems according to the invention is optionally provided in addition to the usual visual communicative possibilities by means of bidirectional acoustic communication. The use of external sources of information via a server and the merger or networking of several of the optoelectronic systems according to the invention are also possible.

When operating the optoelectronic system, the light coupled out of the observation beam path of an imaging optics for the micro, macro, near and far ranges is transmitted into the beam path for determining the spectral properties of one or more objects contained in the scene to be imaged, with the aim of obtaining the spectral information to determine and evaluate.

Unknown spectral properties of the object are measured and classified. The spectral identification can be used for identical replication, that is to say for the storage of the data relating to the spectral properties obtained at several different locations and synchronization of these data sources.

The invention will be explained in more detail with reference to embodiments. The accompanying drawings show in

1 the optoelectronic system according to the invention in a first embodiment, consisting of a first, designed as a spectrometer intent assembly and a second module with the functions of a smartphone, in the camera of the smartphone only the coming of an object light passes, the spectrometer intent has happened

2 the schematic diagram of a second embodiment, consisting of a spectrometer intent, a smartphone having two cameras, an internal light source for object illumination and means for referencing the illumination light,

3 the schematic diagram of a third embodiment, formed from a spectrometer-intent and a smartphone with a camera, wherein in the imaging optics of the smartphone directly from the object coming light and at the same time separated from it light, which has passed the spectrometer intent, and wherein in the Imaging optics incoming light is separated within the smartphone or directed together on an image sensor,

4 the schematic diagram of a fourth embodiment, consisting of two spectrometer intentions and a smartphone having two cameras, with an external natural or artificial light source for scene or object lighting is provided in the imaging optics of a first camera directly from the object coming light and Light that has passed the first spectrometer intent, while in the second camera only light enters which has passed the second spectrometer intent,

5 the schematic diagram of a fifth embodiment, consisting of a spectrometer intent and a smartphone having two cameras, both cameras in contrast to the examples according to 2 and 4 have the same light incidence direction, in the first camera light coming directly from the object passes, and in the second camera only light passes that has passed the spectrometer intent.

6 the schematic diagram of a sixth embodiment, consisting of a smartphone with a camera and an external spectrometer, wherein in the camera and in the spectrometer each coming directly from the object light,

7 the schematic diagram of another embodiment, consisting of a smartphone with a camera and an integrated spectrometer, wherein in the camera and the spectrometer each coming directly from the object light passes.

1 shows the basic structure of a first embodiment of the optoelectronic system according to the invention. A multichannel prism spectrometer, designed in the form of a spectrometer intent 1 , for example, has three channels here 2 . 3 and 4 with which simultaneously three different spectra of an object 11 coming measuring light beam 5 can be measured. Essentially, the spectrometer comprises light columns S2, S3 and S4 assigned to the individual channels, lens groups with collimator function L1.2, L1.3 and L1.4, lens groups L2.2, L2.3 and L2.4 with telescope function and a prism P as a dispersive optical element.

In 1a , A plan view of the said and symbolically indicated functional elements of the spectrometer, are the adjacent in the plane of the channels 2 . 3 . 4 recognizable. 1b shows the same representation in a side view, in which the channels 2 . 3 . 4 Covering each other, lying one behind the other.

When operating this arrangement, the measuring light enters the light columns S2, S3, S4 simultaneously, passes through the channels 2 . 3 . 4 , occurs via an interface 5 from the spectrometer intent 1 from and subsequently into a camera which, for example, essentially comprises an objective O, a zoom lens Z and a position-resolving image sensor B. The camera is preferably part of a smartphone 6 ,

For the purposes of the present invention, the term "smartphone" is to be understood as meaning devices which combine the functions and functional components of a mobile telephone with a PDA (Personal Digital Assistant). They are equipped with at least one digital camera, GPS and WLAN receiver and internal power source, and in addition to the processor have several data memories, for example a very fast but volatile main memory (RAM), a non-volatile NAND flash memory for operating system and programs. and optional removable media, eg in the form of mass storage. In addition to the operating system, the user can install application programs. The synchronization of data with external data storage and data processing equipment is possible.

In the camera of the smartphone 6 arrives in this embodiment of the optoelectronic system according to the invention as the measuring light exclusively from the object 11 coming light that the spectrometer intent 1 happened. The image generation and the spectral analysis are carried out immediately in time. In contrast, the spectral analysis or the measurement of the three different spectra is carried out simultaneously using three different areas B1, B2 and B3 of the image sensor B. Alternatively, the detection of the different spectra with separate sensors is possible and is within the scope of the invention.

2 shows an embodiment of the optoelectronic system according to the invention, consisting of a dual-channel spectrometer intent 1 and a smartphone 6 , In addition, the smartphone points 6 two cameras 7 and 8th on. The cameras 7 . 8th have 180 degrees to each other offset and thus opposite directions of light incidence. Both cameras 7 . 8th are commercially equipped with the necessary for generating static or moving images of a scene or a single object optical components and beam paths. Furthermore, the optoelectronic system according to the invention in this embodiment is provided with an internal light source 9 fitted.

The camera 7 is the spectrometer intent 1 upstream, the camera 8th is used for image generation. That from the light source 9 coming light is over a first ray path 10 as illumination light on an object to be imaged and spectrally analyzed 11 and at the same time over a second ray path 12 as reference light by means of mirror 13 in the spectrometer intent 1 and this in the camera below 7 directed. The object 11 incoming light passes as measurement light over a beam path 14 in the spectrometer intent 1 and at the same time as an imaging light over a ray path 15 , for example by means of mirrors 16 and 17 redirected to the second camera 8th , Imaging and spectral analysis are provided simultaneously.

The optoelectronic system in an embodiment according to 3 consists of a spectrometer intent 1 and a smartphone 6 with a camera 7 , To illuminate the object 11 Ambient light is used. The object 11 incoming measuring light passes over a beam path 18 in the spectrometer intent 1 and subsequently into the camera 7 and at the same time as an imaging light over a ray path 19 directly into the camera 7 , Here, the beam paths for spectral analysis and the beam paths for image formation run separately, and the image generation and the spectral analysis are carried out simultaneously by means of a flat image sensor. The light entering the imaging optics is separated within the smartphone or directed together at the image sensor.

4 shows another embodiment in conjunction with a smartphone 6 which in turn has two cameras 7 and 8th having opposite light incidence directions. The camera 7 is a first spectrometer intent 1 upstream, which serves for spectral analysis, while the camera 8th a second spectrometer intent 20 upstream, which is used for referencing the illumination light. To illuminate the object to be spectrally analyzed 11 is a natural or artificial light source 21 intended.

The illumination light is over a ray path 22 through the spectrometer intent 20 into the camera 8th and at the same time over a ray path 23 on the object 11 directed. The object 11 coming measuring light is comparable to 3 , over a ray path 18 through the spectrometer intent 1 through to the first camera 7 and at the same time over a ray path 19 directly into the camera 7 directed. The beam paths for spectral analysis and the beam paths for image generation run separately, the image generation and the spectral analysis are carried out simultaneously by means of a flat image sensor, of which separate surface sections are used for spectral analysis and image generation.

The schematic diagram of the optoelectronic system according to 5 shows a further embodiment with a smartphone 6 that two cameras 7 and 8th having. Both cameras 7 . 8th have here - unlike in the examples after 2 and 4 - no opposite directions of light. To illuminate the object 11 Ambient light is used. The object 11 The coming measuring light is over a ray path 18 through the spectrometer intent 1 in the first camera 7 and at the same time over a ray path 19 in the second camera 8th directed. The beam paths for spectral analysis and the beam paths for image generation run separately, and the image generation and the spectral analysis are carried out simultaneously by means of a flat image sensor or two flat image sensors. In the case of using an image sensor, separate area sections are used for spectral analysis and image generation. If two image sensors are used, one of them serves the spectral analysis, the second of the image generation.

Out 6 shows an embodiment in which the optoelectronic system according to the invention from a smartphone 6 with a camera 7 and an external spectrometer 24 is formed. To illuminate the object 11 is an artificial light source 25 intended. The illumination light is over a ray path 26 on the object 11 directed. About a ray path 18 the light coming from the object enters the spectrometer 24 , over a ray path 19 the light coming from the object reaches the camera 7 , the coupling of the spectrometer 24 to the smartphone 6 via cable 27 ,

7 shows the schematic diagram of another embodiment, here consisting of a smartphone 6 with a camera 7 and an integrated spectrometer 28 , About the ray path 18 the light coming from the object enters the spectrometer 28 , over the ray path 19 the light coming from the object reaches the camera 7 , The beam paths for spectral analysis and the beam paths for image generation run separately, and the image generation and the spectral analysis are carried out simultaneously by means of a flat image sensor or two flat image sensors. In the case of using an image sensor, separate area sections are used for spectral analysis and image generation. If two image sensors are used, one of them serves the spectral analysis, the second of the image generation.

In all embodiments, the weight of the optoelectronic system according to the invention is a maximum of 1 kg, so that in this respect, the conditions of a hand-held device are met.

• – Farbsensoren für Monitore, Drucktechnik, Video- bzw. Digitalprojektoren zur Korrektur der Farbwerte, oder
• – Farbsensoren für Bauanwendungen zur Farbermittlung von Flächen, Objekten, usw. und nachfolgenden kontrollierten Farbabstimmungen bei diverser Applikationen.

The use of the optoelectronic system according to the invention is carried out, for example, for the purpose of color determination in connection with

• - Color sensors for monitors, printing technology, video or digital projectors to correct the color values, or
• - Color sensors for building applications for color determination of surfaces, objects, etc. and subsequent controlled color matching in various applications.

• – Pulsoxymetern auf Reflexionsbasis,
• – Photometern für Point-Of-Care(POC)-Anwendungen,
• – nasschemische Kopplung zu Photometern,
• – forensische Anwendungen,
• – fernoptischen Erkundungen in Kopplung mit Spektralmessungen,
• – mikrooptische Beobachtungen in Verbindung mit Spektralmessung,
• – militärischen Anwendungen, z.B. Sichtbarmachung von Kampfstoffen, Tarnungen usw.,
• – Erfassung von Lebensmittelinformationen bezüglich Sicherheit, Zustand, Nährwert usw.,
• – agrarwissenschaftlichen Anwendungen wie Chlorophyllbestimmung, Pflanzenzustand, Bonitur, Bodeninformationen usw., wobei sich aus den Reflexionseigenschaften von Vegetation und Bodenformationen z.B. Aussagen über Vegetationsindizes und den Gesundheitsstatus der Pflanzen ableiten lassen,
• – Hautschutzmeldern zwecks Warnung bei gesundheits-schädigender UV-Einstrahlung anstelle der bekannten Systeme auf Basis von Intensitätsmessungen. Das Spektrometer basierte Messsystem wird zur Unterscheidung der UV-A, UV-B und UV-C Anteile und zur getrennten Auswertung der Einzel- und Summensignale über den Zeitverlauf genutzt,
• – Thermografiemeldern, auch verbunden mit Thermometer-Applikationen,
• – Brand- und Rauchmeldern, wobei die Flammen aufgrund ihrer charakteristischen Frequenz und spektralen Strahlung detektiert werden. Brandgase werden durch den Vergleich eines Bildes oder Spektrums ohne und desselben Bildes mit Rauch erkannt. Dies kann auch im NIR-Bereich erfolgen, um den Nutzer nicht zu beinträchtigen. Die Alarmmeldung erfolgt vorzugsweise vor Ort akustisch oder über Funk.

The data collected will continue to be used for ingredient determination in connection with

• - reflection-based pulse oximeters,
• - photometers for point-of-care (POC) applications,
• Wet-chemical coupling to photometers,
• - forensic applications,
• - remote-optical explorations coupled with spectral measurements,
• - micro-optical observations in conjunction with spectral measurement,
• - military applications, eg visualization of warfare agents, camouflage etc.,
• - collection of food information regarding safety, condition, nutritional value etc.,
• - agronomic applications such as chlorophyll determination, plant status, scoring, soil information, etc., which can be derived from the reflection properties of vegetation and soil formations, for example, statements on vegetation indices and the health status of plants,
• - Skin protection detectors for the purpose of warning of health-damaging UV radiation instead of the known systems based on intensity measurements. The spectrometer-based measuring system is used to distinguish the UV-A, UV-B and UV-C components and to separate the evaluation of the individual and sum signals over time,
• - Thermography detectors, also connected to thermometer applications,
• - Fire and smoke detectors, where the flames are detected due to their characteristic frequency and spectral radiation. Fire gases are detected by comparing an image or spectrum without the same image Smoke detected. This can also be done in the NIR area so as not to affect the user. The alarm message is preferably carried out on site acoustically or by radio.

An essential component of the inventive idea is the link with metadata such as time, location, angle of view, the high degree of mobility and the coupling in the feedback as an augmented reality system.

A particularly advantageous application consists in the route guidance of the user when searching for objects with specific spectral properties within arbitrary scenes.

LIST OF REFERENCE NUMBERS

1

Spectrometer intent

2

channel

3

channel

4

channel

5

interface

6

Smartphone

7

camera

8th

camera

9

light source

10

ray tracing

11

object

12

ray tracing

13

mirror

14

ray tracing

15

ray tracing

16

mirror

17

mirror

18

ray tracing

19

ray tracing

20

Spectrometer intent

21

light source

22

ray tracing

23

ray tracing

24

external spectrometer

25

light source

26

ray tracing

27

electric wire

28

internal spectrometer

B2, B3, B4

Areas on image sensor

L1.2, L1.3 L1.4

lens groups

L2.2, L2.3; L2.4

lens groups

O

lens

P

prism

S2, S3, S4

light column

Z

zoom optics

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2010/0309454 A1 [0004]
• DE 19528855 A1 [0045]

Cited non-patent literature

• "Cell Phone Based Platform for Biomedical Device Development and Education Applications; Zachary J. Smith at all; Center for Biophotonics Science and Technology, University of California Davis, Sacramento, California, USA [0002]

Claims (21)

Miniaturized optoelectronic system, comprising - optical components and beam paths for generating static or moving images of a scene or a single object ( 11 ), - optical components and optical paths for determining the spectral properties of one or more objects contained in the scene ( 11 ) or the individual object ( 11 ), - at least one image sensor as an optoelectronic transducer, - electronic components designed to process the output signals of the image sensor, - an information output unit for displaying results in association with the objects ( 11 ), and - means for powering the electronic components - wherein an embodiment of the system is provided in the form of a hand-held device. Miniaturized optoelectronic system according to claim 1, wherein - The image generation and spectral analysis are provided simultaneously or temporally successive, wherein - The respective beam paths separated or run together. Miniaturized optoelectronic system according to claim 1 or 2, wherein the means for determining the spectral properties are designed as multi-channel spectrometer, preferably for the determination of spectra simultaneously for several wavelengths of the measuring and / or illumination light simultaneously with the image formation. Miniaturized optoelectronic system according to claim 3, - equipped with a hyperspectral multichannel spectrometer, and - designed to analyze image data obtained with up to 250 different color channels in the wavelength range from ultraviolet to long-wave infrared, - Wherein the use of a wavelength range detecting silicon based image sensor and / or the use of a plurality of monochromatic, a predetermined wavelength range detecting individual sensors is provided. Miniaturized optoelectronic system according to claim 4, in which - the generation of two- or three-dimensional images of the scene or of the object ( 11 ), and - the determination of the spectra with reference to one or more individual lines, to the area or to the volume of the respective object ( 11 ) is provided. Miniaturized optoelectronic system according to claim 5, wherein the generation of three-dimensional images on the basis of the TOF = Time Of Flight principle is provided. Miniaturized optoelectronic system according to one of the preceding claims, in which the means for imaging and spectral analysis - housed within a common housing, or - housed in separate modules and coupled to each other via interfaces, wherein the means for image forming a dispersive optics, preferably optical grating ( 5 ), Prism or filter. Miniaturized optoelectronic system according to claim 7, - Designed to use the natural light as lighting and measuring light, or - Equipped with an artificial source of illumination and measuring light, where - For coupling the measuring light in the beam paths for determining the spectral properties free-beam optics or optical fibers are present. Miniaturized optoelectronic system according to claim 8, comprising a fluorescence-exciting light-emitting light source. Miniaturized optoelectronic system according to claim 8 or 9, wherein the artificial light sources are connected to a drive circuit for influencing the radiation intensity, the wavelength and / or the polarization of the emitted light. Miniaturized optoelectronic system according to one of the preceding claims, equipped with means for generating instructions for action for the user - as a result of the evaluation of the determined spectral properties, or - as a result of additional linking of these evaluation results with others, the scenes and objects ( 11 ) information. Miniaturized optoelectronic system according to claim 11, equipped with means for guidance of the user in the search for objects ( 11 ) with certain spectral properties within arbitrary scenes. Miniaturized optoelectronic system according to one of the preceding claims, equipped with means for wireless integration into a communication network, in particular designed for image and sound transmission, for transmitting measurement results in external data storage and / or or for retrieving information from external data memories, preferably reference values for calibrations of at least the optical imaging or spectral analysis means. Miniaturized optoelectronic system according to claim 13, wherein the calibration of the intensity and / or the wavelength of the illumination and / or measuring light is provided. Miniaturized optoelectronic system according to one of the preceding claims, wherein the information output unit has an LED or OLED display. Miniaturized optoelectronic system according to one of the preceding claims, wherein the means for image formation are designed as a light field camera system and both the detection of the position and intensity of the light incident on the image sensor and the detection of the direction of incidence is provided. Miniaturized optoelectronic system according to one of the preceding claims, in which a pulsed illumination or measuring light source is provided and the sequence in the recording of moving images is synchronized with the switch-on frequency of the light source. A miniaturized optoelectronic system according to any one of the preceding claims, wherein the image sensor is connected to a drive for temporarily merging adjacent pixels into pixel blocks to achieve higher photosensitivity. Miniaturized optoelectronic system according to one of the preceding claims, equipped with acceleration sensor, gyroscope, inclinometer and designed for integration in the Global Positioning System (GPS). Miniaturized optoelectronic system according to claim 19, wherein the determination of the angle of incidence of the measurement or illumination light on an object to be imaged ( 11 ) is provided in relation to the optical axis of the imaging optics by means of GPS and inclinometer. Miniaturized optoelectronic system according to one of the preceding claims, designed for the classification of imaged objects ( 11 ) based on their spectral properties.

Images