WO2006100209A2 - Field programmable synapses array and method for programming the same - Google Patents

Abstract

The invention relates to a field programmable synapses array (FPSA) for integration into a neuronal network and to a method for programming the FPSA. The aim of the invention is to provide a modular analog neuronal integrated system which comprises flexible yet very complex blocks of neuronal networks for recognition and classification problems. The inventive method combines a plurality of network topologies on an integrated array. The array can be controlled and structured from the exterior and, once configured, is a self-learning classification system that operates autonomously. The FPSA is characterized by an optimized array of connections (synapses) that are optimized in themselves and that are implemented on an integrated system having modular structures as a 'sea of synapses' having no topology, said structure forming a generic network of dendrites and axons. For the teaching the array, a linear function, a sigmoid function, a square function and a Gaussean function is used either individually or in combination.

Description

 description

Field Programmable synapse Array and method for its

programming

Technical environment

The invention is a field Programmable Synapse Array (FPSA), an intelligent processor and a chip solution for the integration of a neural network and a method for programming the FPSA.

Artificial neural networks are models of brain function. They try to simulate brain cell complexes in structure and function and thus to achieve a viable simulation of complex thinking processes. They provide a computational model for information processing and are thus an alternative to computational computer models. These are largely based on Von Neumann architecture. In the literature, the nature of neuronal information processing is shown in great detail u.a. in [Rojas, R .; Neural Networks, Springer Verlag, Berlin and others. 2002], [Zell, A .; Simulation of neural networks; Oldenbourg-Verlag, Vienna, 1994].

Learning and parallelism of processing are among the most important

Characteristics of neural networks. Fault tolerance and generalization capability are often achieved, but depend on the choice of network model, network size, and encoding processing information (input and output patterns).

There are several learning approaches for neural networks: supervised, unsupervised and amplifying.

In supervised learning, the error between calculated and desired output is minimized. In the supervised learning phase, an output vector (e.g., symptoms, measurements or images) is assigned an output vector. When using a neural network as a classifier, the training takes place exclusively with the patterns to be classified. Unlearned patterns and faults are classified as unknown in later use.

Unsupervised learning is learning without well-known goal values. Thereby different things can be learned. Popular are the automatic classification (clustering) or the compression of data for dimensionality reduction. It is the optimization of weights based on criteria that the network itself has.

The learning algorithm of reinforcing learning adapts weights based on maximizing a reward, or minimizing punishment, which is pronounced on the basis of the calculated output becomes.

A problem with the use of neural networks is the rather complex determination of the optimal network structure and the parameters, such as output functions of the neurons, threshold, minimum, maximum, learning rule, learning rate and number of learning steps. Therefore, many tasks employ expensive conventional methods (e.g., statistical or rule-based methods), although the use of neural networks would be simpler, faster, and cheaper.

In addition to the actual algorithmic task is still the requirement of implementation. When a given neural structure and topology is found for a given task, it is important in the embedded implementation requirements to find appropriate target hardware. Here the hardware implementations of neural networks play a role.

In examining existing neurohardware solutions, it is clear that the digital realizations make up the bulk of commercial products. This is due to the fact that digital circuitry is easier to master than analog. On the other hand, the design tools for digital circuit technology are much more convenient and more automated than comparable tools for analog circuit development. Due to the longer development cycles in analogue compared to digital technology, the development costs are correspondingly higher.

Analog circuit technology is particularly for the implementation of certain

Functions of the neural network much more effective than its digital counterpart. These include multiplication and activation functions. It is to be expected that in addition to the digital implementations, the analog approaches will also produce powerful realizations of neural networks in the future.

Previous implementations have shown that with the analog implementation variant of multilayer perceptions with a hidden layer, information processing speeds in the reproduction phase from the input to the output of the network of less than 5 μs are achieved. The non-adaptive analog chip from INTEL, the ETANN, requires 8 μs for a reproduction. The weight accuracy is 6 bits.

The patent US 6513023 Bl is about the implementation of an analog neural network in the form of an MLP. The back propagation algorithm is implemented with on-chip learning. For the neuron, a special circuit is presented, with which the sigmoid function and its derivative can be mapped. The weights are charged analogously as charges in capacities and must be refreshed regularly. Learning takes place under external supervision.

Disadvantage of the invention is that the implementation is fixed. A scaling is only possible with a new design. The implementation is limited to a network with back propagation. There is no other algorithm usable.

In the patent application WO 2003015027 A2, the learning is purely analog, the weight storage takes place during the learning phase analog capacitively in a capacitor and in the working phase locally digital in registers behind MDACs to not have to refreshen. Added to this is the global weight storage in a memory block and monitoring, which are done digitally. An external controller is required, so this is not a pure on-chip learning / self-learning implementation. The analog-to-digital conversion is carried out by 16 implemented ADCs line by line on the chip. The structure of the multilayer perceptron is freely selectable on the chip. The BPA with learning is hard-wired. No other algorithms can be used.

From the patent publication DE 693 30 432 T2 a universal neural network computer and supercomputer is known. It is about the implementation of a cellular neural network (CNN), a multi-dimensional grid arrangement of a plurality of identical cells. Each cell acts directly with a given number of cells of r neighboring layers. The r-neighborhood is the "receive radius field." The application determines whether the layers of neighbor cells are 2- or 3-dimensional The basic unit is the cell with linear and non-linear circuit elements. The proposed solution is a CNN which is programmable without hardware components having to be exchanged.

Disadvantage of this publication is that it is a 2- or 3-dimensional lattice structure in which the processors are arranged. It is not a structure with freely selectable network topology. The processors only interact with each other in a finite neighborhood. Thus, the arrangement is limited to a few algorithms.

In JP 2002358503 A is generally the control of scalable analog

Blocks are described by digital switches and digital connection circuits. The digital circuit blocks of each cell are the so-called "programmable devices." The analog circuit blocks can be configured by programming these "devices", i. be connected with each other. This can be used to structure a network. In addition, there are circuits that each exist for configuring the respective block. In addition, there is a memory block each containing the information about the switching state of the switches and the configuration of the respective circuits.

The disadvantage here is that there is a very general circuit of analog and digital circuit blocks for the implementation of signal processing functions. For learning structures, as they are needed for a neural network, a great effort is needed. There are no pre-assigned activation functions that can be selected according to the network topology and neuron types used. The disadvantages of the aforementioned solutions are that

- The application engineer must know the chips to the smallest detail so that he can use the circuits.

- A very large circuit complexity must be operated to integrate these chips in one application.

- Either only one network topology can be implemented or the network topology is not programmable from the outside.

- A very large circuit complexity must be operated to integrate the non-linear activation functions.

Disclosure of Invention Technical Problem

The object of the invention is to provide a modular analog neuronal integrated system which contains both very flexible, as well as very complex blocks of neural networks for recognition and classification problems. The neural structure on the arrangement should be designed in such a way that the largest possible number of network topologies can be implemented.

It is a further object to combine a plurality of network topologies on an integrated arrangement by a method so that the user is able by programming to select a neural network topology. The arrangement should be able to be controlled and structured from the outside and still be a self-sufficient classification system after the configuration, which learns independently. Technical solution

The object is achieved by the field programmable synapse array according to the invention and a method for programming the FPSA, as described in the claims.

The Programmable Synapse Array field is characterized in that an optimized arrangement of optimized connections (synapses) is topologically realized as "sea of synapses" on an integrated system with modular structure, which realizes a generic network of dendrites and axons, whereby processors (neurons) are connected in such a way that they are layered and map a multiplicity of topologies of neural networks. For teaching the arrangement, a linear function, a sigmoid

Function, a quadratic function and a Gaussian function either individually or in combination of several used.

The existing developments can not be compared with the invention shown here. The main advantage of this solution is that after the production of a standard chip release it can be determined which concrete topology (= application) should be implemented.

The neural network on the integrated device is circuitally designed so that the largest possible number of network topologies can be implemented.

For this purpose, the synapses are designed so that they are arranged in a field and can be connected from the outside via programming as far as possible with each neuron.

The arrangement has, in contrast to other conventional neural hardware solutions, a modular structure. Thus both the number of required synapses and the form of the neuron activation functions can be varied. Depending on the learning process and the selected network topology as well as the given patterns, tolerance and selectivity can be directly trained. These properties mean that a large number of embedded solutions for complex classification tasks becomes possible in the first place.

Thus, the requirement for both highly flexible, as well as more complex components for recognition and classification problems is sufficient.

The simultaneous integration of a standard digital interface increases the flexibility and the handling of the Anorgnung. In connection with the increasing acceptance of neural solutions for technical classification problems, the planned product opens up wide application possibilities. Even after production of a chip release, network topologies can be optimized with modern methods (e.g., evolutionary, genetic, and statistical algorithms) and implemented into the previously produced batch. While hitherto errors in individual blocks of the neural network to be implemented lead to discarding the entire chip production, the success of the chip production is no longer dependent on the quality of the implemented network, since defective synapses or neurons can be switched off.

With these characteristics, the development work (of dependencies) can be unbundled and parallelized since

• be able to work in parallel with the modular system concept module developers and reduce the coordination effort.

• the development of network topologies in parallel and independent of the chip Development can take place.

• The application development does not affect the completion of certain chips

(ASICs), but can rely on standardized product releases. With a variable number of processing units, the arrangement has a

Structure that the user can choose himself. With a processing speed in the nanosecond range requires the

Circuit only a fraction of the time, which are the fastest products for the fastest

Need reproduction. The modularity of the network topology results in easier handling by the application engineer. This reduces the inhibition threshold for switching to a neuronal solution. There are novel applications in the areas of evaluation of sensor arrays in the automotive industry, in the printing industry and in the aerospace conceivable.

Brief Description of Drawings The invention will be further described by way of example. The show

Characters:

FIG. 1 Synapse block with register FIG. 2 Neuron functions FIG. 3 Hidden neuron block FIG. 4 Input hidden output neuron block FIG. 5 Perceptron type FIG. 6 Nearest neighbor type FIG. 7 Example for Network Construction FIG. 8 Single-layer perceptron network and synapses with linear or non-linear

Activation function, network consisting of 4 inputs, 16 synapses and 4 outputs Figure 9 Perceptive network with a hidden layer, network consisting of 4

Inputs, 16 hidden synapses, 4 hidden neurons, 16 output synapses and 4 output neurons Figure 10 perceptron network with two hidden layers, network consisting of 4

Inputs, 16 hidden synapses and 4 hidden neurons in the first hidden layer, 16 hidden synapses and 4 hidden neurons in the second hidden layer, 16 output synapses and 4 output neurons Figure 11 Perceptual mesh with three hidden layers, mesh consisting of 4

Inputs, 16 hidden synapses and 4 hidden neurons in the first hidden layer, 16 hidden synapses and 4 hidden neurons in the second hidden layer, 16 hidden synapses and 4 hidden neurons in the third hidden layer, 16 output synapses and 4 output neurons

FIG. 12 shows two parallel perceptron networks with two inputs each, one hidden layer each consisting of four synapses and two neurons and one output layer each consisting of four synapses and two outputs

Figure 13 A first perceptron network with 2 inputs, a hidden layer consisting of 4 synapses and 2 neurons and an output layer consisting of 4 synapses and 2 outputs and a second perceptron network with 2 inputs, a first hidden layer consisting of 4 synapses and 2 Neurons, a second hidden layer consisting of 4 synapses and 2 neurons and an output layer consisting of 4 synapses and 2 outputs

Figure 14 network for easy classification with 3 inputs and 4 outputs

Figure 15 RB F network with 3 inputs, a hidden layer consisting of 6

Synapses and 2 neurons and an output layer consisting of 4 synapses and 2 neurons

Figure 16 winner-take-all network with 3 inputs, 3 outputs and 15 synapses. If all synapse connections are not trained, this network can also be considered a Kohonen feature map.

Figure 17 Hopfield network consisting of 3 inputs, 3 outputs and 6 synapses

Prerequisite for the implementation is a strictly object-oriented (modular and hierarchical) system concept with clearly defined interfaces between the hierarchy and module levels and the instances of a module level, e.g.

- System

- Function blocks network

- Topology (or parallel processor arrangement) segments

- Neuron (or processor)

- dendrite

- axon

- Synapse

This system concept has the advantage that the various module levels can be optimized and standardized independently of each other.

Under this condition, an optimized arrangement of optimized connections (synapses) can initially be realized on the chip topologically as "sea of synapses".

With this arrangement, a generic network of dendrites and axons realized, which processors (neurons) interconnected so that they can be layered and can map a variety of topologies neural networks.

The arrangement is such that after production of the chip, a specific application-specific topology can be "burned" into the chip.

In analogy to FPGAs and FPAAs, this arrangement can be described as FPSA (Field Programmable Synapses Array).

The FPSA circuits based on this uniform technology can be produced in different scaling stages for different fields of application and thus produce a diversified portfolio of standard products that can be tailored to the respective application.

For the teaching of the chip, a method is developed, with which a selbststän ended fitting the neural network is possible. The reproduction path can be fully integrated. In this variant, the weights are learned externally on a computer and then loaded completely.

The modular chip is a neural classifier that learns independently. The

However, learning is supervised by a trainer during supervised learning. In the case of uncontrolled learning, i.a. Find similarities and assign them to corresponding clusters.

The dimensions of this unit allow for the first time the use of different topologies neural networks in a sensor-related system for mobile and autonomous applications and devices. This simpler and faster implementation of a neural network on the basis of such a chip is intended to replace previous implementation variants and artificial neural networks u.a. significantly simplify decision-making in connection with sensors and transducers.

Properties of the network:

general structural variability, d. H. variable algorithm and scalable mesh size (limit: number of synapses)

- variable number of hidden layers possible (max 4)

- Several networks can be implemented in parallel and independently of one another. The signal transmission between neurons, synapses, inputs and outputs is effected by differential currents. [0067] Synapses All synapses in the synapse array have the same structure and implement the same

Functions (Figure 1). These are the storage of a weight value in

Form of a digitized voltage by storage in digital registers or

RAM cells. On the other hand, in the synapse the multiplication of the stored weight value with the input value applied to the synapse (w * x). If the weight value is stored digitally, a DAC must be implemented in the synapse. The conversion of the analog weight values into digital bit patterns is done by a network of switched:

1. capacities,

2. resistors or

3. Power sources.

The synapses are addressable so that weight values can be written in or read out.

The functional special form Biassynapse is constructed exactly like the synapse and can also be used as such.

The synapses are arranged in blocks of four each. Between the upper and lower as well as between the right and left two four four-wire cables are arranged. The inputs and outputs of the synapses can be:

1. variable programming of transistor gates or

2. Fixed programming by fuses

[0074] Any connection can be made to these lines. However, the two differential input and output lines of a synapse may only be wired to one line at a time. However, several differential input and output lines of different synapses on the same cable pairs can be wired.

In order to make the programming, a register is provided in each synapse block, in which the switch position is stored coded.

Neurons

There are different neuron forms in the network. In general, however, they have the task of calculating the calculation of a function value from the sum of the signals present at its input.

Two functions (FIG. 2) are provided for the neurons. These are:

1. the linear function,

2. the sigmoid function,

3. the quadratic function and

4. the gaussian function.

The sigmoid function is the standard activation function in addition to the linear function. Both are topologically provided in the hidden layers and the source layer. The sigmoid function is used in particular with the multilayer perceptron.

The quadratic function can be used only in the hidden layers, while the Gaussian (Nearest Neighbor) function only in the output layer The function curves can each be set via various parameters (eg sigmoid function - slope can be changed via working current).

Of the four functions, either one selected or a combination of several may be made.

The hidden neuron functions can also assume a synaptic function task in special cases. Should, for example In the case of the nearest-neighbor network, the function for calculating the Euclidean distance (w-x) is executed by a synapse, this can be implemented here by the sum of (w -2 * w x + x).

The value w is generated in a biassynapse, the value -2 * wx in a general synapse and the value x ² in a hidden neuron and the

Outputs on a node added to (wi-xi) ² .

The neuron functions are combined into blocks. The wiring is done as with the synapses by programming a register. For the hidden neuron blocks (FIG. 3), one block each is implemented with the linear, one with the sigmoid function and one with the quadratic function together with a synapse. The synapse can be e.g. be used as a bias or feedback synapse, which is required in fed-back networks. These neuron blocks are placed on the left, right and top edges of the mesh.

At the bottom are blocks with three inputs and two neurons each. These output-hidden neuron blocks (Figure 4) can optionally be treated as neurons of the hidden or output layer. One neuron contains the linear and the sigmoid function and the other neuron the linear and Gaussian functions. These two neurons do not contain connections to bias or feedback synapses. But a switchable driver is implemented in them, which makes it possible to control external loads without the need for additional circuitry between the chip and connected actuators. At the same time the influence of additional sources of error on the output signals of the arrangement is avoided because the signal flow between the sensor and the actuator is kept extremely short. There is no analog / digital or digital / analog conversion. In addition, no further drivers and converters are needed in the optimal case.

FIG. 5 shows the perceptron type. In the perceptron type, the input signals are weighted (multiplied by a weight value). This is followed by a summation of the weighted input signals. This corresponds mathematically to the scalar product. After scalar formation, the sum is transferred by a special function (linear, sigmoid, jump-shaped).

With this type, all possible networking topologies can be realized then, for example, become the so-called multilayer perceptron.

FIG. 6 shows the nearest neighbor type. In the nearest neighbor type, the difference between the input signal and the weight is formed and then squared, and the sum of the squares is then formed.

If one subtracts the square root of the sum, this corresponds to the calculation of the distance (Eukliedische distance) between applied signal and stored weight vector (hence the name "nearest neighbor") .This is not crucial for the function of the network but would have the advantage that this value is geometrically interpretable in 1 to 3-dimensional vectors, such as in the 3-dimensional color space.

With the "Nearest Neighbor type" opens (by some additions) a whole series of network variants, which are known as:

- Radial Basis Functions Networks (RBF)

- Generalized Regression Networks (GRNN) Probabilistic Neural Networks (PNN)

- Self-Organizing Maps (SOM) (also Kohonen Feature Maps)

- Learning Vector Quantization Networks (LVQ)

It must be emphasized that behind the different network variants does not always hide another architecture, but rather that a training method or usage is responsible for the name.

Perceptronnets - a layer and linear transfer function

This is the simplest case of a perceptronnetzes. The interesting thing is that the weights can be calculated analytically. There is exactly one global minimum. One can use this network form, e.g. for linear color space transformations such as for sensor correction or for conversions between different color monitor types.

Perceptronnets - A layer and non-linear transfer function

The weights can be calculated conditionally analytically. With a simple iterative minimization method one finds a global minimum.

[0095] Perceptronnets - multiple layers

The determination of the weights for the task to be solved can only be done with a statistical iterative minimization method (learning). There are generally a variety of local minima. Therefore, the success of weight determination is not predictable but random. It is also not possible to specify the appropriate topology of meshing for a particular task because there are no analytical relationships between task type and topology properties. The minimization process may require a great deal of iterations and may require a lot of time (depending on the computing power). In principle, with multilayer perceptron networks classification and

Function approximation tasks are solved. It should be noted, however, that a real classification becomes possible only through an additional layer, namely from the output vector of the network by e.g. a maximum determination the class emerges.

The quality of a function approximation is difficult to verify, since not the entire (multi-dimensional) signal range can be traversed. It is therefore possible that the response of the network to a particular input signal unpredictably deviates from the desired behavior.

Nearest Neighbor Type - Simple Classifier

At this point, only two simple representatives are to be listed by way of example, on which the essential properties become clear.

This network has only one layer and no special transfer function. It can be directly evaluated the sum at the output. In the weights, the so-called prototypes of a class are stored. The calculation of the network produces at the output of the network a vector which represents the (quadratic) distances of the input signal to the stored class prototypes. By determining the minimum output signal, the appropriate class is found. Ideally, the minimum is zero because the input signal and prototype are identical.

This network type is very well suited for classification tasks in which class prototypes can be specified and where no further re-mapping of the output signals is necessary. A well-known example is the classification of colors.

It is advantageous that is determined by the topology (number of inputs and outputs) exactly what the chip and what does not. It can therefore be said in advance whether a task is solvable or not.

Nearest Neighbor Type RBF Network

The RBF network has two layers. The first layer is fully wired to the input, as in the simple multilayer perceptron. However, the weighted sums are calculated in a special function that has a bell shape. Thus, the layer provides for test patterns that are outside the training pattern only minor activities, since only areas that lie within the respective bells lead to activities of the neurons. The second layer of the RBF network is of the perpe- trone type with a linear output function. Therefore, the weights of this layer can be calculated analytically, so that no statistical iterative minimization process is required.

The second layer can realize various tasks (anything that a single-layered network can do). One function is "mapping". transition vector of the first layer assigned a specific vector at the output of the second layer, for example, to define classes.

Furthermore, the second layer may allow the RBF network to be used for general functional approximations.

Advantageous here is the property that inherently no unpredictable

Reactions of the network can arise to unknown input signals, since the network interpolates only between the stored nodes.

The hidden neuron functions can also assume a synaptic function task in special cases. Should, for example In the nearest neighbor network, the function for calculating the Euclidean distance (w-x) is performed by a synapse, this can be implemented by the sum of (w -2 * w x + x).

The value w is generated in a biassynapse, the value -2 * wx in a general synapse and the value x ² in a hidden neuron and the

Outputs on a node added to (wi-xi) ² .

The synapse neuron array has a strictly modular design to produce a nearly arbitrary shape and size of a neural network. Figure 7 shows a proposal for setting up the FPSA. The structure is designed in such a way that as many conceivable as possible realistic network topologies can be implemented in hardware.

[Ol l i] The neurons are grouped in blocks. The wiring is done as with the synapses by programming a register. For the hidden neuron blocks, one block each is implemented with the linear, one with the sigmoid and one with the quadratic function together with a synapse. The synapse can be e.g. be used as a bias or feedback synapse, which is required in fed-back networks. These neuron blocks are placed on the left, right and top edges of the mesh. At the bottom are blocks with four neurons that perform the sigmoid function. One can do this e.g. each split into three input and one output neuron.

[0112] In FIG

[0113]

Table 2

SAl ₅ I Synapse S - synapse, A - synapse in block (A, B, C or D),

Column number, line number

INLl input neuron

BNR bias neurons

HNLl hidden neuron

dramatic activation function), arrangement in the block (L - left, R - right, O - above), column or row number, contains linear, sigmoidal and quadratic activation function

OHNAl Output Hiddenne OHN - Output hidden neuron, A - I. Neuron in the block uron (contains linear and sigmoid activation function), B - 2. Neuron in the block (contains linear and Gaussian activation function), column number

In the example, a network with 64 synapses, 12 biassynaps, 16 hidden neurons, 12 input neurons, 4 output neurons and 3 bias neurons is shown.

A practical network could contain 1088 (1024 synapses + 64 biassynapses) synapses and 131 (48 input neurons + 64 hidden neurons + 16 output neurons + 3 bias neurons) neurons.

[0116] perceptron networks

In perceptron networks, the input signals are multiplied by a weight value. This is followed by a summation of the weighted input signals. This corresponds mathematically to the scalar product. After scalar formation, the sum is transferred by a special function (linear, sigmoid, jump-shaped).

With this type, all possible cross-linking topologies can be realized, which are then e.g. become the so-called multilayer perceptron.

In the following, by way of example, simple representatives of perceptron networks are to be listed.

[0111] Perceptron network with a weight layer and linear transfer function

This is the simplest case of a perceptronnetzes. The interesting thing is that the weights can be calculated analytically. There is exactly one global minimum. This network shape can be used e.g. for linear color space transformations such as for sensor correction or for conversions between different color monitor types.

A practically realized perceptron network with a weight layer and linear transfer function for the cited FPSA example is shown in FIG. The network has 4 inputs (INAl, INA2, IN A3, IN A4), a weight layer consisting of 16 synapses (SAl ... 4,1 ... 4) and 4 outputs (OHNAl, OHNA2, OHNA3, OHN A4). The network can be operated with or without bias neurons and synapses. The bias neurons can also be used as additional input neurons. In the neuron blocks HNRl, HNR2, HNR3 and HNR4, the linear activation is performed. The synapses SD 1,1, SD2,2, SD3,3 and SD4,4 are for signal redirection only. There is no weighting in them. Single layer perceptron network with nonlinear transfer function

The weights can be calculated conditionally analytically. With a simple iterative minimization method one finds a global minimum.

A practically realized perceptron network with an output layer and non-linear transfer function for the cited FPSA example is shown in FIG. The network has 4 inputs (INAl, INA2, IN A3, IN A4), a weight layer consisting of 16 synapses (SAl ... 4,1 ... 4) and 4 outputs (OHNAl, OHNA2, OHNA3, OHN A4). The network can be operated with or without bias neurons and synapses. The bias neurons can also be used as additional input neurons. In the neuron blocks HNR1, HNR2, HNR3 and HNR4, nonlinear activation is performed. The synapses SD 1,1, SD2,2, SD3,3 and SD4,4 are for signal redirection only. There is no weighting in them.

[0126] Perceptron networks with multiple hidden layers and linear or nonlinear transfer function

The determination of the weights for the task to be solved can only be done with a statistical iterative minimization method (learning). There are generally a variety of local minima. Therefore, the success of weight determination is not predictable but random. It is also not possible to specify the appropriate topology of meshing for a particular task because there are no analytical relationships between task type and topology properties. The minimization process may require a great deal of iterations and may require a lot of time (depending on the computing power).

In principle, classification and function approximation tasks can be solved with multilayer perceptron networks.

A perceptron network for the cited FPSA example with one, two, or even three hidden layers and with also optionally usable in the individual layers linear or nonlinear transfer function is thus feasible. Depending on the topology requirements, the networks can be operated with or without bias neurons and synapses. The bias neurons can also be used as additional input neurons.

A network with 4 inputs (INAl, INA2, IN A3, IN A4), a hidden layer consisting of 16 synapses (SAl ... 4,1 ... 4) and 4 neurons (HNRl, HNR2, HNR3, HNR4) and an output layer consisting of 16 synapses (SB 1 ... 4,1 ... 4) and 4 outputs (OHNAl, 0HNA2, 0HNA3, OHN A4) is shown in FIG. 4 inputs (INAl, INA2, IN A3, IN A4), a first hidden layer consisting of 16 synapses (SAl ... 4,1 ... 4) and 4 neurons (HNRl, HNR2, HNR3, HNR4) , a second hidden layer consisting of 16 synapses (SB 1 ... 4, 1 ... 4) and 4 neurons (HNOl, HNO2, HNO3, HNO4) and an output layer consisting of 16 synapses (SCl ... 4,1 ... 4) and 4 outputs (OHNAl, OHNA2, OHNA3, OHNA4) has the network, which shows Figure 10. The activation is carried out in the neuron blocks HNL1, HNL2, HNL3 and HNL4. The synapses SD 1,1, SD2,2, SD3,3 and SD4,4 are for signal redirection only. There is no weighting in them.

An example of a network with 4 inputs (INAl, INA2, IN A3, IN A4), a first hidden layer consisting of 16 synapses (SAl ... 4,1 ... 4) and 4 neurons (HNRl, HNR2, HNR3, HNR4), a second hidden layer consisting of 16 synapses (SBl ... 4,1 ... 4) and 4 neurons (HNOl, HNO2, HNO3, HNO4), a third hidden layer consisting of 16 synapses ( SCl ... 4,1 ... 4) and 4 neurons (HNL1, HNL2, HNL3, HNL4) and an output layer consisting of 16 synapses (SDL ... 4,1 ... 4) and 4 outputs (OHNAl, OHNA2, OHNA3, OHNA4) is Figure 11 again.

The three examples already show how great the possibilities of variation are only for perceptron nets, which are possible with the cited FPSA example.

Parallel arrangement of multiple perceptron networks with one or more weight layers and linear or non-linear transfer function

In order to be able to process a plurality of sensor signals which are completely independent of one another, but without having to operate the same amount of hardware over several times, the FPSA offers the possibility of implementing parallel networks which are completely independent of one another. The topologies used can also be completely different.

FIG. 12 shows an implementation example for the cited FPSA example, if two topologically identical networks with 2 inputs each (INA1JNA2 or IN A3, IN A4), each with a hidden layer consisting of 4 synapses (SAI..2, 1..2 or SA3 ... 4,3 ... 4) and 2 neurons (HNRl, HNR2 or HNR3, HNR4) each with an output layer consisting of 4 synapses (SBl ... 2,1 ... 2 or SB3 ... 4,3 ... 4) and 2 outputs (OHNAl, 0HNA2 or 0HNA3, 0HNA4). The networks can be operated with or without bias neurons and synapses, or linear or nonlinear activation function. The bias neurons can also be used as additional input neurons.

If one wishes to operate two networks with different topologies in the cited FPSA example, FIG. 13 shows a possible implementation variant. There is a first network with 2 inputs (INAl, INA2), a hidden layer consisting of 4 synapses (SAl ... 2,1 ... 2) and 2 neurons (HNRl, HNR2) of an output layer consisting of 4 synapses (SB 1 .. .2,1 ... 2) and 2 outputs (OHNAl, OHNA2) and a second network with 2 inputs (IN A3, IN A4), a first hidden layer consisting of 4 synapses (SA3 ... 4,3 .. .4) and 2 neurons (HNR3, HNR4), a second hidden layer consisting of 4 synapses (SB3 ... 4,3 ... 4) and 2 neurons (HNO3, HNO4) and an output layer consisting of 4 synapses (SC3 ... 4,3 ... 4) and 2 outputs (OHNA3, OHNA4). For the second network, the activation is carried out in the neuron blocks HNL3 and HNL4. The synapses SD3,3 and SD4,4 are only for signal redirection. There is no weighting in them. Both networks can be operated either with or without bias neurons and synapses or linear or nonlinear activation function. The bias neurons can also be used as additional input neurons.

[0138] Nearest Neighbor Type

In the Nearest Neighbor type, the difference between the input signal and the weight is formed and then squared, and the sum of the squares is then formed.

[0140] If one subtracts the square root of the sum, this corresponds to the calculation of the distance (Euclidean distance) between adjacent signal and stored weight vector (hence the name "Nearest Neighbor") .This is not decisive for the function of the network but would have the advantage that this value is geometrically interpretable in 1 to 3-dimensional vectors, such as in the 3-dimensional color space.

Networks from the "Nearest Neighbor" open up a whole series of network variants.

By way of example, only two simple representatives of nearest-neighbor-type networks are to be listed here by way of which the essential properties become clear.

[0143] Simple Classifier

This network has only one layer and no special transfer function. It can be evaluated directly the sum of the Euclidean distances at the output. In the weights, the so-called prototypes of a class are stored. The calculation of the network produces at the output of the network a vector which represents the quadratic distances of the input signal to the stored class prototypes. By determining the minimum output signal, the appropriate class is found. Ideally, the minimum is zero because the input signal and prototype are identical.

[0145] This type of network is very well suited for classification tasks in which class Prototypes can be specified and where no further re-interpretation (mapping) of the output signals is necessary. A well-known example is the classification of colors.

A practical network for the cited FPSA example for easy classification is shown in FIG. The classifier has 3 inputs (INAl, INA2, INA3) and 4 outputs (OHNAl, OHNA2, OHNA3, OHNA4). In the simple classification, first the difference between the input signals and the weights is formed. This happens in the hidden neuron blocks (HNRl ... 4, HNLl ... 4, HNOl ... 4). The bias value (BNR, BNL, BNO) is set to -1. This results in the weight value (wi) in the synapses (SRI ... 4, SLl ... 4, SOl ... 4), which are stored in the hidden neuron blocks (HNRl ... 4, HNLl ... 4, HNOl .. .4) are multiplied by -1 (-wi). The input signal (xi) is added to this value (xi-wi). Then this sum is squared ((xi-wi) 2). The sum of the squares is then formed at the downstream input hidden output neuron blocks (OHNA1, 0HNA2, 0HNA3, 0HNA4) (sum ((xi-wi) 2)). The synapses SBI ₅ I, SB2,2, SB3,3, SB4,4, SCI ₅ I, SC2,2, SC3,3, SC4,4, SDl ₅ I SD2,2 SD3,3 and SD4,4 are only for signal routing. There is no weighting in them.

[0147] RBF network

The RBF network has two layers. The first layer is a hidden layer like the simple multilayer perceptron. However, the weighted sums are calculated in a special function that has the shape of a bell curve. Thus, the layer provides for test patterns that are outside the training pattern only minor activities, since only areas that lie within the respective bells lead to activities of the neurons. The second layer of the RBF network is of the perpe- trone type with a linear output function. Therefore, the weights of this stage can be calculated analytically, so that no statistical iterative minimization process is required.

The second layer can realize various tasks. One function is "mapping", where each output vector of the first layer is assigned a particular vector at the output of the second layer to define classes, for example, and the second layer allows the RBF network to use general function approximations leaves.

[0150] The advantage here is the property that, in principle, no unpredictable

Reactions of the network can arise to unknown input signals, since the network interpolates only between the stored nodes.

An implementation example of the cited FPSA could be a 3-input RBF network (INAl, INA2, INA3), a hidden layer consisting of 6 synapses (SB1,1, SB2,2, SC 1,1, SC2,2, SD 1,1, SD2,2) and 2 neurons (OHNBl, OHNB2) and an output layer consist of 4 synapses (SB 1,4, SB2, 4, SC 1.4, SC2, 4) and 2 neurons (OHNAl OHN A2), which Figure 15 shows. The network operates like a simple classifier with a downstream output layer. The two hidden neurons (OHNBl, OHNB2) work with the Gaussian function as activation. The bias neurons (BNL, BNO, BNR) are used to reference the weight values stored in the synapses SL1, SL2, SO1, SO2, SR1 and SR2. The synapses SAl ₅ I, SA1.2, SA2.1, SA2,2, SA3.1, SA3,2, S Al, 4 and SA2,4 as well as the neuron blocks HNL4 and HNR4 are used for signal deflection. In these blocks, no weighting or activation is made.

[0152] Feedback networks

So far, only network topologies without feedback have been considered. Processing by propagation is from the input layer to the output layer. In the case of feedback networks, on the other hand, the states of the neurons must be recalculated until the network has converged to an idle state in which no change in the activation states results. A stable state is therefore determined by the input, the weight matrix and the threshold values of the neurons.

[0154] In the following, three typical representatives of feedback networks will be briefly presented.

[0155] Winner take AU (WTA)

In competitive learning (competition learning), the units of the network co-operate with each other to create the "right" to input an issue, and only one is allowed to deliver one issue while preventing others from doing so.

In the case of unsupervised learning, the n-dimensional inputs are processed with as many arithmetic units as how clusters are to be determined. For three clusters, a network of three competitive units is needed.

A practically realizable WTA network for the cited FPSA, which shows Figure 16, has 3 inputs (INAl, INA2, INA3) and 3 outputs (OHNAl, 0HNA2, 0HNA3). The network is single-layered and fully networked with 9 synapses (SAl ₅ I ... 3, SA2,1 ... 3, SA3.1 ... 3). The outputs of each neuron (HNR1, HNR2, HNR3) are additionally switched to the inputs of all other neurons with 6 synapses (SB1,2, SB1,3, SB2.1, SB2,3, SB3.1, SB3,2) , The synapse blocks SD 1,4, SD2,4 and SD3,4 are not used as synapses, but are only used to connect the outputs of the neurons to the FPSA outputs.

[0159] Kohonen Feature Maps

Kohonen feature maps work similar to WTA networks. The difference is however, that the output is not limited to one unit and the number of clusters is not known before learning. The clustering is thus generated during learning.

An example of how to implement a Kohonen network with 3 inputs and 3 outputs in the cited FPSA example is shown in FIG. The network is exactly the same as in the WTA example. However, the illustrated feedback connections through synapses SB1,2, SB1,3, SB2,1, SB2,3, SB3,1 and SB3,2 may not be formed during teaching.

[0162] Hopfield networks

Hopfield networks are based on the following assumptions:

- The network consists of a single layer of n neurons.

The n neurons are totally networked with each other, i. Each neuron has a connection to every other neuron (feedback, recursion).

- No neuron is directly connected to itself (no immediate feedback).

The network is symmetrically weighted, i. the weight of the bond between neuron i and neuron j is equal to the weight of the connection between neuron j and neuron i.

- Each neuron is assigned a linear threshold function as an activation function.

- Input is the usual weighted sum.

Hopfield networks are therefore single-layered neural networks, which have only indirect feedbacks between each two different nodes i, j (i ≠ j) of the network, but no direct feedback to the same node. All connections between two neurons are symmetric, i. wij = wij. This can also be interpreted as meaning that there is only one bidirectional line between two neurons.

The Hopfield network converges - under certain conditions - to a quiescent state after a finite time. When a part of the output lines is fed back to the input, the corresponding part of the output pattern y may contribute to the input pattern x, y = x, i. Input of the neurons = output of the neurons (autoassociation).

Providing an incomplete input pattern to this feedback system in which such pattern pairs are stored results in a correspondingly incomplete output pattern. Its correct areas may be sufficient to partially supplement the missing portions when returning to the entrance. The improved output pattern is again presented as input to the network so that the system will respond with a further improved output. Every output of a Neurons act by the feedback on the inputs of all neurons back.

The Hopfield network in Figure 17 with 3 inputs (INAl, INA2, INA3) and 3 outputs (OHNAl, OHNA2, OHNA3) represents another implementation possibility of a topology in the FPSA. The network is single-layered with 6 synapses (SB 1,2, SB 1,3, SB2,1, SB2,3, SB3,1, SB3,2) are fully fed back. Each neuron (HNR1, HNR2, HNR3) has a connection to every other neuron. No neuron, however, is directly connected to itself. The synapse blocks SAI ₅ I, SA 2, 2, SA 3, 3, SBI ₅ I, SB 2, 2 and SB 3, 3 are not used as synapses, but merely serve to switch the inputs or outputs of the neurons to the FPSA input. or outputs.

[0168] Application

With the integration of multiple neural network topologies, other learning algorithms can be used as well. The combination of modularity, speed and a large number of processing units makes the chip a novelty, which enables many applications in sensor processing. The applications for the invention are primarily in the field of sensor technology (sensor signal processing, adaptation), in industrial automation, as well as other areas in which very fast control functions are involved. This mainly affects systems with changing environmental conditions. Here, a learning ability for event states is required (teach-in functionality). The training takes place via a host computer with a special training software.

Due to the technical parameters of the product, since many inputs are available, parallel evaluations of multi-sensor arrays in one component are made possible.

The sensor market is one of the fastest growing segments in the entire microelectronics market. Based on standardized interfaces, it is possible to easily link sensors to networks.

This trend arises from qualitatively new requirements for the sensor signal processing, for example:

- increasing miniaturization

- growing chip integration

- intelligent procedures

- decentralized processing

- increasing real-time requirements

- Increasing complexity of detected states / situations

The predominant proportion of added value thus lies next to the sensor elements themselves, above all in the subsequent sensor-near signal processing. The sensor elements are getting smaller, better, more integrated, more robust and cheaper. Conceivable applications can be found in the main areas of wireless communication, automotive industry and industrial process automation. Here, for example, in the application fields to solve tasks such as channel equalization, classification, and adaptive control, this list is not exhaustive.

Claims

claims

Field programmable analog synapse array consisting of synapses, which are realized on an integrated system with a generic network of dendrites and axons, characterized in that an optimized arrangement in itself optimized connections analogously implemented synapses topologielos as "sea of synapses" on a integrated system with modular structure, whereby processors (neurons) are connected in such a way that they are layered and map a multiplicity of topologies of neural networks, whereby the signal flow between sensor and actuator is kept extremely short and no analog / digital or digital / Analog conversion takes place.

Field programmable synapse array according to claim 1, characterized in that the synapses are designed so that they are arranged modularly in a field and are connected from the outside via programming with each neuron, so that both the number of required synapses with variable algorithm and scalable network size as well as the shape of the neuron activation functions vary, allowing for a variable number of hidden layers and implementing multiple networks of different or similar topology in parallel and independently of each other.

Field programmable synapse array according to one of claims 1 or 2, characterized in that the synapses are arranged in blocks of four and between the upper and lower and between the right and left two synapses four double-wire lines, the inputs and Outputs of the synapses by variable programming of transistor gates or fixed programming by fuses on these lines are arbitrarily switched.

Field programmable synapse array according to one of claims 1 to 3, characterized in that the synapses in the synapse array are the same structure and addressable and realize the same functions.

Field programmable synapse array according to claim 4, characterized in that in the synapses, the storage of a weight value in the form of a voltage by storage in the C-unit (poly-poly-cap or gate-cap) or in a digital register and the multiplication of the stored weight value with the input value applied to the synapse (w * x).

Field programmable synapse array according to one of claims 1 to 5, characterized in that for programming in each synapse block, a register is provided, in which the switch position is stored coded.

Field programmable synapse array according to one of claims 1 to 6, characterized in that the neurons, the linear, sigmoidal, Gaussian and quadratic functions are provided, which are individually selected or a combination of several is made.

Field programmable synapse array according to claim 7, characterized in that the linear and sigmoidal functions are the standard functions and are provided in the input, hidden and output layers.

Field programmable synapse array according to claim 7, characterized in that the quadratic function is used only in the hidden layer.

Field programmable synapse array according to claim 7, characterized in that the Gaussian function is used only in the output layer.

Field programmable synapse array according to one of claims 7 to 9, characterized in that for the hidden neuron blocks in each case a block with linear, one with the sigmoid and one with the quadratic function are implemented together with a synapse.

Field programmable synapse array according to one of claims 7 to 11, characterized in that the neuron blocks are placed on the left, right and upper edge of the network, wherein at the bottom blocks with input neurons and two output Hiddenneuronen are arranged.

Field programmable synapse array according to claim 12, characterized in that special output hidden neuron blocks are arranged, which contain a combination of either linear and sigmoid function or linear and Gaussian function.

Field programmable synapse array according to one of claims 12 to 13, characterized in that special output hidden neuron blocks are arranged which contain a switchable function that can drive external loads.

Field programmable synapse array according to one of claims 1 to 14, characterized in that the neurons are combined into blocks and the wiring is carried out by programming a register.

Field programmable synapse array according to one of claims 1 to 15, characterized in that a plurality of networks can be implemented in parallel and independently of each other.

Field programmable synapse array according to one of claims 1 to 16, characterized in that the neurons and the synapses are each pre-structured and these prestructured circuits are interconnected via switches.

Field programmable synapse array according to one of claims 1 to 17, characterized in that the signal transmission between neurons, synapses, inputs and outputs is effected by differential currents.

Field programmable synapse array according to one of claims 1 to 18, characterized in that it contains a switchable driver, which makes it possible to control external loads, without operating additional circuitry effort between the arrangement and connected actuators.

Method for Programming a Field Programmable Analog

Synapse arrays, which are realized from synapses that are implemented on an integrated system with a generic network of dendrites and axons, are characterized in that an arrangement of optimized connections of analogously implemented synapses are topologically arranged in a modular manner as "sea of synapses" and externally connected to each neuron via programming so that both the number of synaptic syndromes required with variable algorithm and scalable network size and the shape of the neuron activation functions vary, allowing a variable number of hidden layers and multiple networks of different or more similar Topology can be implemented in parallel and independently of each other and the teaching of the system by an automatic mechanism with which an independent adjustment of the neural network is possible.

A method according to claim 20, characterized in that for the teaching of the arrangement a linear function, a sigmoid function, a quadratic function and a Gaussian function is used either singly or in combination of several.

A method according to claim 21, characterized in that the sigmoid function and the linear function are provided as standard activation functions in the hidden and output layer.

A method according to claim 22, characterized in that the standard activation functions are used in Multilayerperzeptron.

A method according to claim 21, characterized in that the quadratic function is used only in the hidden layer, wherein the function curves are each set via different parameters.

Method according to one of claims 21 or 24, characterized in that the Gaussian function is used only in the output layer, wherein the function curves are each set via different parameters.

Method according to one of claims 20 to 25, characterized in that the Hiddenneuronenfunktionen take over a synaptic function task by performing the function of calculating the Euclidean distance (w -x) by a synapse by the sum of (w -2 * wx + x), where the value w is generated in a biassynapse, the value -2 * wx in a general purpose synapse and the value x ² in a hidden neuron and the outputs on a node are added to (wi-xi) ² .

A method according to claim 21, characterized in that

Input signals are multiplied by a weight value, then a summation of the weighted input signals (scalar product) and after the scalar transmission of the sum by a special function, in particular by a linear, a sigmoide or a jump-shaped function occurs.

A method according to claim 27, characterized in that with a weight layer and linear transfer function, the weights are calculated analytically and exactly one global minimum exists.

[0029] Method according to claim 27, characterized in that, with a weight layer and non-linear transfer function, the weights are conditionally calculated analytically and a global minimum is found with a simple iterative minimization method.

A method according to claim 29, characterized in that the networks are operated depending on the requirements of the topology with or without bias neurons and synapses and / or the bias neurons are used as additional input neurons. A method according to claim 27, characterized in that one or more weight layers and linear or non-linear transfer function, the weights for the task to be solved with a statistical iterative minimization method are determined, wherein a plurality of local minima exists.

A method according to claim 27, characterized in that with a parallel arrangement of several perceptron networks with one or more weight layers and linear or non-linear transfer function will process a plurality of completely independent sensor signals, the topologies used may be completely different.

A method according to claim 20, characterized in that a difference between the input signal and weight formed, this is then squared and the sum of the squares is formed, wherein the distance (Euclidean distance) is calculated by pulling the square root of the sum and this the calculation corresponds between the applied signal and stored weight vector.

A method according to claim 33, characterized in that only one layer and no special transfer function is used in the directly evaluated the sum of the Euclidean distances at the output and stored in the weights of the so-called prototypes of a class, whereby at the output of the network Vector is formed, which represents the square distances of the input signal to the stored class prototypes, and by a determination of the minimum output signal, the matching class is found.

Method according to claim 33, characterized in that two layers are used, the first layer being a simple weight layer, the sums of which are subsequently calculated in a special function in the form of a bell curve and the second layer is of the perpectron type with a linear output function whose weights are calculated analytically, so that no statistical iterative minimization process is required.

A method according to claim 20, characterized in that feedback networks are used in which the states of the neurons are recalculated until the network is converged into a resting state in which there is no change in the activation states more.

A method according to claim 36, characterized in that a stable state by the parameters input, weight matrix and Thresholds of neurons is determined.

A method according to claim 36 or 37, characterized in that the n-dimensional inputs are processed with as many arithmetic units as clusters are to be determined and only one unit of the network delivers an output and at the same time prevents all others from doing so.

A method according to claim 36 or 37, characterized in that the n-dimensional inputs are processed with as many arithmetic units as clusters are to be determined and the output is not delivered by only one unit and the clustering is generated during learning.

A method according to claim 36 or 37, characterized in that the network consists of a single layer of n neurons which are totally networked with each other, i. Each neuron has a connection to every other neuron, no neuron is directly connected to itself, the network is symmetrically weighted, the individual neurons each have a linear threshold value function assigned as an activation function, and the usual weighted sum is input.

A method according to claim 36 or 37, characterized in that an incomplete input pattern is input to the feedback system and when returning to the input the missing portions are partially supplemented and the improved output pattern is presented to the network as an input again so that the system responded with a further improved output.