Dynamic Capacity Estimation in Hopfield Networks

09/15/2017
by   Saarthak Sarup, et al.
Columbia University
0

Understanding the memory capacity of neural networks remains a challenging problem in implementing artificial intelligence systems. In this paper, we address the notion of capacity with respect to Hopfield networks and propose a dynamic approach to monitoring a network's capacity. We define our understanding of capacity as the maximum number of stored patterns which can be retrieved when probed by the stored patterns. Prior work in this area has presented static expressions dependent on neuron count N, forcing network designers to assume worst-case input characteristics for bias and correlation when setting the capacity of the network. Instead, our model operates simultaneously with the learning Hopfield network and concludes on a capacity estimate based on the patterns which were stored. By continuously updating the crosstalk associated with the stored patterns, our model guards the network from overwriting its memory traces and exceeding its capacity. We simulate our model using artificially generated random patterns, which can be set to a desired bias and correlation, and observe capacity estimates between 93 97 networks in comparison to the static and worst-case capacity estimate while minimizing the risk of lost patterns.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

07/30/2013

Neural Network Capacity for Multilevel Inputs

This paper examines the memory capacity of generalized neural networks. ...
02/05/2013

A Non-Binary Associative Memory with Exponential Pattern Retrieval Capacity and Iterative Learning: Extended Results

We consider the problem of neural association for a network of non-binar...
10/04/2018

A Practical Approach to Sizing Neural Networks

Memorization is worst-case generalization. Based on MacKay's information...
10/27/2019

Zero-Error Capacity of Multiple Access Channels via Nonstochastic Information

The problem of characterising the zero-error capacity region for multipl...
11/14/2012

Memory Capacity of a Random Neural Network

This paper considers the problem of information capacity of a random neu...
11/24/2015

Dynamic Capacity Networks

We introduce the Dynamic Capacity Network (DCN), a neural network that c...
07/24/2014

Convolutional Neural Associative Memories: Massive Capacity with Noise Tolerance

The task of a neural associative memory is to retrieve a set of previous...

1 Introduction

Associative memory storage remains one of the most appealing features of human cognition, enabling complex functions in the hippocampus like efficient memory retrieval and information routing (Levy and Steward, 1979; Stella and Treves, 2011). Unsurprisingly, associative memory storage has also been an important goal of modern neural networks, and low-latency models like Hopfield networks have traditionally been effective solutions to this problem.

Described by Little (1974) and later popularized by Hopfield (1982)

, (Little-)Hopfield networks are a form of recurrent neural networks which consist of

fully-connected McCulloch-Pitts neurons. In addition to this topology, Hopfield networks are typically designed with the constraint that no neuron has a connection with itself and that all connections are symmetric. They have the ability to store binary patterns of the form , and are commonly learned according to the standard Hebb rule, formalized as a sum-over-outer-products. These patterns are then recalled by probing the network with a possibly noisy version of the desired pattern and allowing the network to relax into a low energy state according to its attractor dynamics (Aiyer et al., 1990; Nasrabadi and Choo, 1992; Storkey and Valabregue, 1999).

A common question asked of these networks is their capacity — how many patterns can a -neuron Hopfield network reliably store? Increasing the number of learned patterns introduces additional terms called crosstalk (MacGregor and Gerstein, 1991), which can be intuitively understood as the influence of stored patterns on the network response to a desired pattern. The accumulation of these crosstalk terms results in higher order instabilities in the stored patterns, as they produce spurious states as undesirable fixed points in the -dimensional attractor space (Wu et al., 2012). We explore the mechanics of this behavior in Section 4.1, which enables our effective capacity estimation for the Hopfield network.

Here we define this term, capacity, as the maximum number of stored patterns which can be perfectly recalled when probed with the desired pattern. We consider alternate definitions and their relation to ours in Section 2. Nevertheless, mathematically rigorous analyses have been conducted on this question using a compatible definition of capacity, producing coarse estimates like to more precise expressions like (Amit et al., 1985; McEliece et al., 1987). Unfortunately, these static estimates fair poorly on a number of relevant metrics like accuracy and efficiency, as we observe in Section 5.

Figure 1: Example capacity simulation for Hopfield network. The dark shaded region indicates the efficiency improvement using our proposed dynamic model in comparison to a static estimate with the worst-case correlation and bias. Lightly shaded region shows the minimal efficiency difference between our dynamic model and the true maximum efficiency.

Later analyses by Löwe have extended prior work by incorporating the possibility for bias (Löwe, 1999) or correlation (Löwe, 1998) in the stored patterns. Their theoretical results propose a reduction in memory capacity as either bias or correlation diverge from the ideal i.i.d. random pattern case, and experimental simulations testing these hypotheses support their claims (Wilson and Osborne, 1998).

Since these static capacity models do little in the way of accounting for naturally varying patterns, Hopfield networks using a standard Hebb learning rule are rarely effective in practice. Designers hoping to perfectly recall all stored patterns are then forced to drastically underestimate the capacity for the worst-case scenario, as we depict in Figure 1 above. This is of particular importance in VLSI design, as hardware designers intending to use Hopfield networks as a form of content-addressable memory use an impractically large amount of memory to store the network’s weight matrix relative to the network’s information capacity (Draghici, 2000; Theocharides et al., 2004; Verleysen et al., 1989).

In place of the standard static estimates for memory capacity defined in Section 2, we present a novel dynamic technique for monitoring the capacity of a Hopfield network in real time. Our estimation model is a function of both the current state of the network, as represented by the network’s weight matrix, and the incident pattern to be stored. By testing the current network’s response to a new pattern, we calculate a fraction of the total crosstalk in our system, allowing us to observe the network’s distance to its capacity. This behavior is more quantitatively analyzed in Section 4.2. We then compare our results on the same metrics used to analyze static estimates and note an increase from an average of 53% accuracy to an average of 94% accuracy using our dynamic capacity model. Moreover, we observe a doubling of memory efficiency when compared to the static models that assume worst-case capacity estimates.

2 Related work

Hopfield and other early researchers like Amit, Gutfreund and Sompolinsky considered a notion of capacity defined for an -neuron Hopfield network by a critical value , such that for , nearly all patterns are memorized, and for , nearly all patterns are forgotten (Hopfield, 1982). By comparing Hopfield networks to physical spin-glass models and assuming i.i.d. random patterns, they conclude with a calculation of and therefore represents the capacity of a Hopfield network (Amit et al., 1985). We find this definition of capacity too coarse for ensuring minimal memory loss and thus is not considered any further in this work.

The question of capacity was brought up again by McEliece, Posner, Rodemich and Venkatesh with the more precise definition that capacity is the maximum number of i.i.d. random patterns which can be stored and perfectly recalled. They extend their analysis to considering initial probes with patterns at a distance of Hamming distance away from one of the patterns, where . Their exact derivation is not useful here but can be found in (McEliece et al., 1987). Their final expression takes the following form:

(1)

Löwe broke from the assumption of i.i.d. random patterns to consider the effect of pattern bias (Löwe, 1999) or correlation (Löwe, 1998) on the capacity. He defined a bias for the -th value of the -th pattern by

. He modeled both semantic and spatial correlations as homogeneous Markov chains, defining the correlation

as . For both cases, biased and correlated, Löwe expresses the capacity as , where is defined based on the bias or correlation of the stored patterns.

(2)
(3)

The definition of capacity employed in this paper is the maximum number of stored patterns that a Hopfield network can retrieve when probed with the desired pattern. This definition is compatible with the definitions used by McEliece et. al. and by Löwe, and are used as baselines to which we compare our dynamic estimation model in Section 5. For the static estimate by McEliece et. al., our definitions are equivalent to the case where and thus we use the expression . For the estimate by Löwe, we recognize his lower bound for the parameter and thus we use both its lower bound for the appropriate value of or .

3 Random pattern generation

Since we examine the behavior of Hopfield networks on artificial data, it is important to precisely define how we generate our test patterns. Each of the patterns is represented by an

-dimensional column vector

, where each element is chosen from the set . In the i.i.d. case, each element is chosen independently of all other elements for and

, and with equal probability for either

or .

In order to produce data with bias and/or correlation, we rely on techniques introduced by Löwe (Löwe, 1999). We introduce bias by setting the probability that a particular element is either or , formalized as and thus . In order to introduce correlation, we define a parameter and consider semantic correlations between successive patterns as Markov chains. We then combine these two parameters based on a method described by Clopath et al. (2012). Explicitly, we define the probabilities as follows:

(4)
(5)
(6)
(7)

This approach allows us to consider the input patterns taking the form of independent two-state Markov chains. We can then reason about the correlation between successive patterns and quantify the crosstalk in the Hopfield network as a function of the parameters and .

3.1 Pattern statistics

Each two-state chain can be represented by a transition matrix where elements contain the probability of moving from state to state in one step.

(8)
(9)
(10)

We can extend this transition matrix to steps as

. Recognizing the eigenvalues of

as 1 and , we can calculate

(11)

A useful quantity in our later analysis of crosstalk in Section 4.1 is the correlation between two stored patterns separated by steps, quantified as . As a result, we are interested in describing as well as the expected value . Writing as a function of , , allows us to simplify the expression for the expected value to . We calculate this function as follows:

(12)

We can gain a more intuitive understanding of these equations by evaluating their order for increasing values of . This corresponds to thinking about the correlation between two patterns separated by units of time. Since these correlations are not dependent on our bias , we leave a constant and analyze the behavior as a function of our correlation .

It becomes evident then that for , the expected value of the term decreases for larger distances . This behavior fits our understanding of the Markov property where the future state is dependent on the current state, and thus for increasing separations in time the successive patterns grow decreasingly correlated.

4 Dynamic capacity estimation model

As we will analyze in this section, the fundamental source of memory loss in Hopfield networks arises from an accumulation of crosstalk. Since crosstalk is dependent on the patterns being stored, which can vary drastically in bias and correlation, traditional static approaches to estimating capacity are often inaccurate at predicting the capacity of the network for any particular trial. Taking a dynamic approach to measuring the crosstalk in real time enables us to adapt our estimate of the network’s capacity to the patterns being stored and halt its operation before stored memories are lost.

4.1 Quantitative analysis of crosstalk

Consider an -neuron Hopfield network with synapses stored in an weight matrix . We plan to store patterns of length as for , and represent the network after storing the -th pattern as . Initially, . After the first pattern is stored, we can write the weight matrix as , and after patterns, we write the matrix as:

(13)

Suppose we now attempted to recall all patterns. We calculate their activation and note that, at least to first order recurrence in the network, .

We notice that along with the desired component we also recover crosstalk terms which correspond to the contribution of other memories on . We also notice that the crosstalk varies based on the neuron , and so we define corresponding to the total crosstalk affecting the -th pattern at neuron .

(14)

Modeling our input data as a Markov chain allowed us to express the expected value of the term as . We can treat

as a random variable and express its expected value as

(15)

Knowing that the function decreases for increasing values of based on the correlation of the patterns allows us to use future measurements of and estimate the distribution of crosstalk across all of its terms, as we will show in Section 4.2.

Another notable feature of crosstalk is that it comes in two variants: constructive and destructive. Since the recovered bit depends on the sign of , it is possible that is the same sign as the desired bit, and thus we get that . Thus we are only concerned with the case when . When this occurs, the -th neuron in the network returns the incorrect bit for the pattern and causes the overall network to deviate from its trajectory towards the desired fixed point in and instead settle in an incorrect low energy state.

4.2 Our approach

In order to monitor a Hopfield network’s capacity, suppose we maintain a matrix which contains the destructive crosstalk for the Hopfield network, where we define the elements as . Consider the result of storing an pattern as . When attempting to recover this new pattern through the networks activation to , we calculate . Using this notation, we see the crosstalk for the pattern in all neurons already bundled together as . Thus we can use the following iterative approach to calculate exactly the destructive crosstalk associated with each new pattern , appending a column to our matrix in order to keep track of the new crosstalk:

(16)

However, this calculation alone is not sufficient for monitoring the crosstalk for all patterns, as adding a pattern to the network also introduces crosstalk for the previously stored patterns. After storing the pattern, we must update the crosstalk for all patterns at all neurons with the following quantities.

We can note that the sum of these crosstalk terms becomes . Thus the new quantity we have calculated as the crosstalk for pattern is also the total crosstalk to be distributed across the other patterns.

Using our knowledge of the pattern’s bias and correlation , we know the expected distribution of for all from the calculated quantity . We know that our function , which describes the expected correlation between two patterns separated by units in time, scales like . As a result, we can understand the individual crosstalk terms as follows, though of course the final expressions which our model uses are functions of and .

Updating our destructive crosstalk matrix in this way, written explicitly in Figure 1(b), allows us to iteratively and continuously update the crosstalk for each memory at each neuron. Checking if the network is at capacity thus becomes a simple question of checking if for any and , and allows us operate alongside a Hopfield network as in Figure 1(a) and halt the network before it has exceeded its capacity.

(a)
1 function monitorCrosstalk;
Output : Boolean for under/over capacity
2 if  equals 0 then
3      
4else
5      
6       Calculate row-vector
7      
8      
9 end if
return any
(b)
Figure 2: High level diagram in Figure 1(a) describes how our model interacts with the learning Hopfield network. Figure 1(b) recounts the underlying pseudocode used by our model to update the crosstalk and monitor the network’s capacity.

5 Experiment

5.1 Correctness of our random pattern generator

To verify our pattern generator, we consider bias as and correlation as . We measure the bias of the generated patterns from the mean of each produced pattern, and the correlation from the mean covariance between successive patterns. We note unexpected deviations for highly biased and highly correlated patterns, and thus do not test our model in this regime.

(a)
(b)
Figure 3: Measured bias in Figure 2(a) and measured correlation in Figure 2(b), shown only for patterns with length to avoid redundancy.

5.2 Accuracy on random unbiased, uncorrelated data

To simulate the behavior of our model, we fix our values for network size , pattern bias , and correlation . We then generate random patterns according to the desired bias and correlation, and begin the process of simulating our estimator. Each incident pattern is fed through our estimator and stored in the network as is outlined in Section 4.2. We complete each trial once the estimator flags the capacity and the network crosses its true capacity, allowing us to compare our model’s estimate.

We first look to analyze the behavior of our model on the ideal case of unbiased and uncorrelated data, using and . This case is also applicable for the case when the biased pattern case when we use the zero-centering learning rule. We run 20 trials at neuron counts of , , , , and , corresponding to network sizes ranging from to synapses. We quantify the accuracy of our estimate as the ratio of our estimate to the true capacity , and compare our results on this metric to those from the static model proposed by McEliece et. al. as .

Size Model Accuracy
Dynamic
Dynamic
Dynamic
Dynamic
Table 1: Capacity simulation results with for unbiased, uncorrelated patterns

Results

We can immediately recognize the approximately improvement in capacity estimate from the static model to our proposed dynamic model in Table 1 above. Interestingly, the model proposed by McEliece et. al. does not seem to degrade for larger values of , suggesting the general expression is likely a good approximation for a Hopfield network’s capacity. Nevertheless, tuning this expression for different values of would not take into account natural variations in bias and correlation, and thus is not a practical solution for accurately predicting the network’s capacity. Instead, our dynamic model showed the most promising results, maintaining an accuracy between about across all tested network sizes.

5.3 Efficiency on biased or correlated data

Allowing for variance in the bias or correlation of our stored patterns introduces a notion of the worst-case capacity estimate. To quantify the reduction in memory efficiency when utilizing the worst-case capacity estimate, we define efficiency as the ratio between the estimated capacity

to the number of synapses .

(17)

This metric is analogous to the ratio of the amount of information stored to the number of bits needed to store the information, and the worst-case capacity estimate where corresponds to a reduction in efficiency.

We test our dynamic model in several non-ideal regimes corresponding variable levels of bias and correlation. Specifically, we test our model on values of and , and on and . We compare our efficiency both to the maximum efficiency, , and to Löwe’s static capacity model . We choose for the static model based on the worst case value of bias or correlation, and . For brevity we report a sampling of these tests for a network size of , though they are representative across our previously examined range of . For patterns with high bias and correlation, we are unable to faithfully compare our model’s efficiency to the static model’s efficiency because the static model vastly overestimates the memory capacity, leading to a rapid destruction of all previously stored patterns and an effective efficiency of 0%.

Results

It is clear from our results in Table 2

that our dynamic model has a higher efficiency than the static model. Moreover, our model’s efficiency is within one standard deviation of the maximum efficiency, which fully utilizes the capacity of the Hopfield network. More precisely, we report an average of two-fold improvement on the efficiency as compared to the static model.

Model Efficiency Maximum Efficiency
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Table 2: Capacity simulation results in Hopfield network for biased or correlated patterns

6 Discussion

In this work, we have presented a new model for dynamically recognizing the memory capacity of Hopfield networks. Our model runs alongside a learning Hopfield network and dynamically concludes at an accurate capacity estimate that minimizes memory loss and maximizes efficiency. Moreover, our model solves this problem for increasingly large networks and is moderately robust to variations in bias and correlation. We recognize a computational and memory tradeoff when using our model, as it has complexity from calculating and requires memory to store crosstalk for all patterns in each neuron. However, these costs are on the same order as the process for storing and recalling a pattern and the memory needed to store the weight matrix.

Our dynamic model enables enormous improvements on prior static methods by increasing the effective capacity of an neuron Hopfield network while ensuring near-perfect recall of all stored patterns. It also opens up possibilities for system-level improvements and algorithms such as reducing the computational overhead by reusing the pattern-recall pathway for calculating the new crosstalk, and selectively accepting or rejecting patterns based on their impact on the network’s crosstalk. We feel these are powerful and interesting applications of our dynamic capacity estimation model. Our model can have significant implications for modern designers creating low-power, online learning systems, and repositions Hopfield networks as practical and efficient forms of associative memory.

References

  • Aiyer et al. [1990] Sreeram V.B. Aiyer, Mahesan Niranjan, and Frank Fallside. A theoretical investigation into the performance of the Hopfield model. IEEE Transactions on Neural Networks, 1(2):204–215, 1990.
  • Amit et al. [1985] Daniel J. Amit, Hanoch Gutfreund, and Haim Sompolinsky. Storing infinite numbers of patterns in a spin-glass model of neural networks. Physical Review Letters, 55(14):1530, 1985.
  • Clopath et al. [2012] Claudia Clopath, Jean-Pierre Nadal, and Nicolas Brunel. Storage of correlated patterns in standard and bistable purkinje cell models. PLoS Comput Biol, 8(4):e1002448, 2012.
  • Draghici [2000] Sorin Draghici. Neural networks in analog hardware—design and implementation issues. International Journal of Neural Systems, 10(01):19–42, 2000.
  • Hopfield [1982] John J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982.
  • Levy and Steward [1979] William B. Levy and Oswald Steward. Synapses as associative memory elements in the hippocampal formation. Brain Research, 175(2):233–245, 1979.
  • Little [1974] William A. Little. The existence of persistent states in the brain. Mathematical Biosciences, 19(1-2):101–120, 1974.
  • Löwe [1998] Matthias Löwe. On the storage capacity of Hopfield models with correlated patterns. The Annals of Applied Probability, 8(4):1216–1250, 1998.
  • Löwe [1999] Matthias Löwe. On the storage capacity of the Hopfield model with biased patterns. IEEE Transactions on Information Theory, 45(1):314–318, 1999.
  • MacGregor and Gerstein [1991] R.J. MacGregor and George L. Gerstein. Cross-talk theory of memory capacity in neural networks. Biological Cybernetics, 65(5):351–355, 1991.
  • McEliece et al. [1987] Robert J. McEliece, Edward C. Posner, Eugene R. Rodemich, and Santosh S. Venkatesh. The capacity of the Hopfield associative memory. IEEE Transactions on Information Theory, 33(4):461–482, 1987.
  • Nasrabadi and Choo [1992] Nasser M. Nasrabadi and Chang Y. Choo. Hopfield network for stereo vision correspondence. IEEE Transactions on Neural Networks, 3(1):5–13, 1992.
  • Stella and Treves [2011] Federico Stella and Alessandro Treves. Associative memory storage and retrieval: involvement of theta oscillations in hippocampal information processing. Neural Plasticity, 2011, 2011.
  • Storkey and Valabregue [1999] Amos J. Storkey and Romain Valabregue. The basins of attraction of a new Hopfield learning rule. Neural Networks, 12(6):869–876, 1999.
  • Theocharides et al. [2004] Theo Theocharides, Gregory Link, Narayanan Vijaykrishnan, Mary Jane Irwin, and Wayne Wolf.

    Embedded hardware face detection.

    In 17th International Conference on VLSI Design, 2004, pages 133–138. IEEE, 2004.
  • Verleysen et al. [1989] Michel Verleysen, Bruno Sirletti, Andre M. Vandemeulebroecke, and Paul G.A. Jespers. Neural networks for high-storage content-addressable memory: VLSI circuit and learning algorithm. IEEE Journal of Solid-State Circuits, 24(3):562–569, 1989.
  • Wilson and Osborne [1998] K.F. Wilson and C.F. Osborne. Effects of pattern bias on retrieval dynamics in 0,1 and -1,1 Hopfield networks. Journal of Physics A: Mathematical and General, 31(9):L171, 1998.
  • Wu et al. [2012] Yue Wu, Jianqing Hu, Wei Wu, Yong Zhou, and KL Du. Storage capacity of the Hopfield network associative memory. In Fifth International Conference on Intelligent Computation Technology and Automation (ICICTA), 2012, pages 330–336. IEEE, 2012.