Approximate Hubel-Wiesel Modules and the Data Structures of Neural Computation

by   Joel Z. Leibo, et al.

This paper describes a framework for modeling the interface between perception and memory on the algorithmic level of analysis. It is consistent with phenomena associated with many different brain regions. These include view-dependence (and invariance) effects in visual psychophysics and inferotemporal cortex physiology, as well as episodic memory recall interference effects associated with the medial temporal lobe. The perspective developed here relies on a novel interpretation of Hubel and Wiesel's conjecture for how receptive fields tuned to complex objects, and invariant to details, could be achieved. It complements existing accounts of two-speed learning systems in neocortex and hippocampus (e.g., McClelland et al. 1995) while significantly expanding their scope to encompass a unified view of the entire pathway from V1 to hippocampus.



page 7


Invariance of visual operations at the level of receptive fields

Receptive field profiles registered by cell recordings have shown that m...

A Sinc Wavelet Describes the Receptive Fields of Neurons in the Motion Cortex

Visual perception results from a systematic transformation of the inform...

The Gabor-Einstein Wavelet: A Model for the Receptive Fields of V1 to MT Neurons

Our visual system is astonishingly efficient at detecting moving objects...

Parameter estimation of platelets deposition: Approximate Bayesian computation with high performance computing

A numerical model that quantitatively describes how platelets in a shear...

A Review of In-Memory Space-Efficient Data Structures for Temporal Graphs

Temporal graphs model relationships among entities over time. Recent stu...

Geometric Analysis of the Conformal Camera for Intermediate-Level Vision and Perisaccadic Perception

A binocular system developed by the author in terms of projective Fourie...

Path-specific Effects Based on Information Accounts of Causality

Path-specific effects in mediation analysis provide a useful tool for fa...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

An associative memory can be seen as a data structure wherein stored items are mapped to labels. Or equivalently, items are grouped into sets

, each consisting of all the items sharing the same label. If the items are input feature vectors and the sets correspond to episodic memories then this is a standard view of (part of) the hippocampus. A generalization of an associative memory that is appropriate in any metric space maps items—represented as vectors of distances to stored exemplars—to numbers. The output of this map may be interpreted as the item’s degree of membership in a particular

. If the items are feature vectors, the exemplars are simple cell receptive fields, and the are complex cell receptive fields, then this is a standard view of primary visual cortex.

In this article, we explore the implications of adopting a unified view of hippocampus and the ventral visual pathway based around this data structure analogy. This is a fairly radical idea in light of much of the literature in these areas. The computational and experimental neuroscience communities have historically treated perception and memory as almost entirely separate fields. Episodic memory is critically supported by the hippocampus and the rest of the medial temporal lobe (MTL) [5, 6], thus models of this type of declarative memory have tended to concentrate on these structures e.g., [9]. Conversely models of the ventral visual processing stream typically span from V1 to inferotemporal cortex (IT) e.g., [10]

, the last unimodal area in the ventral visual pathway. However, IT cortex is in fact only one synapse away from perirhinal cortex (PRC) in the MTL

[11]. Yet despite this close anatomical proximity, researchers focusing on either side of this divide are separated by a wall of terminological, methodological and cultural differences. Here we argue that from a computational perspective this barrier is actually largely artificial and arbitrary, and therefore should be broken down.

A major limitation of nearly all computational models of the MTL to date (e.g., [9, 12]) is the simplifying assumptions made regarding the inputs to the memory system. Most models use abstract input features (e.g., arbitrary binary patterns) for reasons of convenience rather than attempting to accurately capture particular properties of the perceptual representations in the afferents to the MTL. Because these abstracted representations cannot be computed from real sensory data, most MTL models are limited to highly simplified situations. The natural place to look for models of perception capable of bridging this gap is the literature on computational models of the ventral stream e.g., [13, 14, 15, 16].

Unified models of neural computation are often motivated by neocortex’s profound plasticity in cases of missing e.g., congenital blindness [17, 18] or rewired inputs [19]

. Under these proposals the differing properties of brain regions arise due to the different inputs they receive, as opposed to their implementing fundamentally different algorithms. Such unified models include convolutional neural networks

[20] and their relatives [21, 13]

and have influenced the current state of the art in machine learning

[22, 23]. However, one should be skeptical they can be straightforwardly extended into the MTL. Unlike the six layered neocortex of the ventral stream, hippocampus is made up of a quite different three-layered structure called archicortex. If there is any point in this extended cortical hierarchy where a different algorithm is most likely to lurk, the transition from entorhinal cortex to hippocampus would be a good bet. The unified view we propose here is mostly on the algorithmic level of analysis [24]. As such, we argue that the brain implements at least two different approximations to the same underlying data structure. The specific approximation that is most suitable will turn out to differ between neocortex and hippocampus.

The rest of this paper proceeds as follows. In section 2 we motivate a particular biologically plausible hierarchy of modules. Then we consider how its connections could be learned from data. In section 3 we consider an idealized learning process. This ideal itself is not biologically plausible, but the new metaphor for neural computation introduced in section 4 motivates several approximations which are plausible. These are described in section 5. Next, we point out that different approximations will turn out to be suitable for different timescales of learning, i.e., fast in hippocampus and slow in neocortex. The final section describes simulations of one model arising from this perspective.

2 A Hierarchy of Associators

It is well-known that the sample-complexity of associative learning can be decreased by incorporating assumptions on the space of hypotheses. One way to do this is to employ preprocessor modules that associate together regions of the state space that ought to be treated the same way. The argument is recursive, from the point of view of the preprocessor module: it too could be learned with lower sample complexity if it had its own preprocessor, and so on. But what is the optimal way to grow out this hierarchy of subordinate associators? How many levels should there be? How many branches per level? Clearly, these answers are determined by the environment itself. Therein lies the fundamental problem: the environment cannot be known in any way but statistically. Since the world is stochastic, the only way to learn what states ought to be associated is to wait until enough information has come in. In this context, there are two arguments for why brains need a two-speed memory system, i.e., a hippocampus and a neocortex.

  1. If a hierarchy of modules must all learn and act at once in a non-stationary world, then the higher level module must always operate on a faster timescale than its subordinates. If it did not, the changing semantics of its inputs would quickly render its own associations useless.

  2. If every stationary aspect of the environment is “filtered out” by the lower modules, the input passed to the associator at the highest level will only contain the information specific to the current timestep. There is therefore no need for the highest level to accumulate more than one timestep before deciding. Thus there is, in principle, no limit to the speed at which it can decide.

In a continuous state-space, an associator clustering its space into regions maps each state to a vector called the signature of . This signature function represents a graded membership to each of classes. Each class is characterized by a particular set , called a template book. For a pooling function , invariant to the order of its arguments, and a similarity function , the -th element of the signature is:


We typically choose to be or , but many other functions are also possible (see [25]). In the case of , the signature can be thought of as the vector of similarities of to the most similar item in each of sets of stored templates. If a hard membership is required, then will assign to one of the classes. Every level of the hierarchy of modules computes its own signature of the input. The inputs to layer are encoded by their signatures at layer .

3 Hubel-Wiesel Modules

The connectivity required to implement a layer of associators is thought to exist in at least one part of the brain. Take to be the response of a V1 simple cell, and all the to be different simple cell receptive fields sharing the same preferred orientation but differing in their preferred position. Then, assuming Hubel and Wiesel’s conjecture for the connectivity of simple (abbr. S) and complex (abbr. C) cells is at least approximately true, is the response of the C-cell that pools that set of S-cells [7].

We call the neural network consisting of one C-cell and all its afferent S-cells a Hubel-Wiesel module abbr. HW-module. We assume that . For a nonlinearity , let t be the set of S-cell responses to . The function could be the sigmoid of the dot product, i.e., as in classical neural networks. However, in most of the present work we use a normalized dot product so we can use the intuition that it measures the similarity between and . An HW-module is a reusable network motif. It appears as an essential architectural element in a wide range of machine learning systems including convolutional neural networks [20, 26], HMAX [13, 27], and nearest neighbor search [28]. One HW-layer consists of many HW-modules. An HW-architecture consists of many HW-layers stacked in a deep hierarchy.

Notice that this notation treats parametric and nonparametric models the same way. In the former case, the templates

would usually be obtained by optimization with respect to a loss function. In the nonparametric case, they can be thought of as training data points (or functions of them). For nonparametric supervised learning, the template book

could be the subset of the training data with label . In the unsupervised parametric case, learning the corresponds to learning “feature pooling” e.g., it could be done by agglomerative clustering [29]. In the unsupervised nonparametric case, any clustering algorithm, or alternatively, temporal-continuity-based associative methods like [30, 31] could be used to assign training examples to template books.

3.1 Invariance and the ventral stream

The different sensory streams have different computational goals. In the case of the ventral stream, the computational goal is to enable visual object recognition—the crux of this problem being that of computing an invariant and discriminative representation for unfamiliar objects. One way to construct such a representation with an HW-architecture is to let the template book be the orbit of a base template under the action of a group . Use to denote a (unitary) representation of an element of (by an abuse notation it can indicate the corresponding group element as well). The -th template book is then . This can be regarded as the outcome of an idealized temporal association process [32, 30, 25].

The orbit is invariant with respect to the group action and unique to each object [25]. For example, the set of images obtained by rotating is the same set of images obtained by rotating . Thus the set of scalar products between and all the is the same as the set of scalar products between and , though their order will be permuted. Since the pooling function is unaffected by permuting the order of its arguments, the output of an HW-module with a group-generated template book is invariant to the action of [25, 33]111Notice also that an HW-module could pool over a subset of the orbit [25]

. This gives a way to model neurons that respond maximally to their preferred feature at any position within a receptive field having some limited spatial extent. The canonical examples are C-cells in V1, i.e., the same cells that motivated the HW-module notion in the first place. A classic convolutional neural net, e.g., in the sense of

[26], is obtained by choosing = translations (or a subset). In that case, each will be a copy of shifted to a different position..

3.2 Binding and multimodal invariance in the medial temporal lobe

In section 2 we argued that an associator’s burden of sample complexity could be lessened by employing a preprocessor consisting of a set of subordinate associators. Priors for wiring up the HW-modules comprising each may be chosen so that particularly useful aspects of the environment are highlighted. However, it is possible that the very fact of being a good subsystem for pulling out some environmental property makes a system worse at pulling out others. For example, information about an object’s identity and its position may conflict in this way. It seems that a system capable of making arbitrary associations, like the hippocampus, is needed in such cases. We propose, in accord with [34], that the hippocampus’s ability to quickly make arbitrary associations makes it likely to be a key player in the binding together of representations from the ventral and dorsal visual streams. It could play a similar role with respect to cross modal representations [35].

4 The Data Structures of Neural Computation

The usual metaphor for a multistage feedforward neural computation is a chain of representation transformations. Each stage, which might correspond to a brain region, e.g., V1, V2, V4, is regarded as a function taking the neural activity vector representing a stimulus in the language of the previous stage as its input and returning to the next stage a transformation of it [21, 36, 16]

. This idea of a transformation cascade has been useful as a description of the ventral stream and as a motivator for computer vision algorithms. However, it may not be rich enough to naturally accommodate the range of phenomena one would like to model in the MTL literature. Now we explore an alternative metaphor for multistage feedforward computation. Rather than the central question being: what is the chain of transformations? Our proposal asks: what are the data structures implemented by cortex at each stage?

An HW-module can be viewed as a data structure. In this view, an HW-module consists of a set of data values and associated operations for accessing and manipulating them. For example, one could be the tuple . INSERT is an abstraction of learning. After obtaining experience with a new stimulus, its representation gets inserted in the correct format to one or more HW-modules. The inference process at test-time is a cascade of QUERY operations. The input stimulus is used to query the first stage HW-modules which return—rather, pass along to query the next stage—a result that depends on the relationship between the input and the stored data. Other operations such as DELETE are also possible, indeed this extensibility is one of the motivations for adopting the data structure interpretation. However, the scope of the present paper will be limited to the two basic operations.

For example, the following describes an HW-architecture implementing a

-way 1-nearest neighbor classifier. Each of the

HW-modules stores a set of data , and comes with two functions INSERT and QUERY. The data held in the -th HW-module is the template book consisting of the set of examples with label : , where the -th template book is the set of examples with label . For a new -labeled example , and an input ,


The predicted category is .

In conjunction with the data structure perspective, nonparametric models like nearest neighbors can capture episodic memories naturally. To remember a specific stimulus, like a phone number, just INSERT

it. It is not as clear how to get such behavior with parametric models. Thus, for the remainder of this paper we restrict our discussion to the nonparametric case.

5 Approximate HW-Modules

So far we have discussed an idealized case, we may call it an exact HW-architecture. Its INSERT operation (2) is not biologically plausible. We propose instead that each stage of the brain’s feedforward hierarchy implements an approximation to it. Different approximation methods may be used in different stages.

5.1 Two biologically plausible approximations

Let and corresponding to the template book . Each template is a row of . Choose to be a normalized dot product. If and are normalized, the vector computed by the S-cells of the -th HW-module is just .

The best rank- approximation of

is its singular value decomposition (SVD)

, where , , and . Let indicate the concatenation of as an extra row of . Thus, if each S-cell stores a row of , the best rank- approximation of the exact HW-module is :


Any online PCA algorithm could be used to update as new data is inserted. The most biologically plausible is the learning rule proposed by Oja as an approximation to the normalized Hebbian rule [37]222

Oja’s rule converges to a solution network that projects new inputs onto the first eigenvector of the past input’s covariance, i.e., onto the first column of

. In the presence of noise, Oja’s rule may also give other eigenvectors. There are also modifications of Oja’s basic rule that find as many eigenvectors as desired [38, 39].. Oja’s rule provides a biologically plausible way to implement INSERT

. However, it generally takes several epochs of looping through the same data items before it converges. Next we discuss another approximation strategy that—while it does not yield the best rank-

approximation—supports rapid insertion of new data.

Since random projections may preserve dot products (as in the Johnson-Lindenstrauss theorem [40]), it is also possible to approximate the S-layer response vector by where is a random matrix satisfying the hypotheses of the Johnson-Lindenstrauss theorem (with ). This approximation will generally be less efficient than the PCA approximation obtained from Oja’s rule. However, it has a fast INSERT operation:


Increasing the rank of with each insertion allows the HW-module to store arbitrary amounts of data without running into the Johnson-Lindenstrauss bound [40]. Alternatively, it might only augment when near the bound.

The dichotomy between these two biologically plausible INSERT operations motivates an interesting conjecture concerning complementary “fast” and “slow” learning systems in the hippocampus and neocortex [8]. The PCA approximation, implemented by Oja’s rule, could operate in cortex while a random-projection-based approximation could be the approximation used in the hippocampus. Since the latter requires the creation of new random vectors, this could be why there is neurogenesis in dentate gyrus and why it is not needed in neocortex.

This conjecture can be seen as a revision of McClelland et al.’s complementary learning systems proposal. However, in our case, the reason for the two learning systems is not to cope with catastrophic interference. Instead, we highlight that cortex, which deals with more constrained tasks, can implement strong priors appropriate for each one. For example, the temporal continuity of object motion can be leveraged toward unsupervised learning of invariant representations

[32, 41]. Hippocampus however, must be able to make arbitrary associations. Thus its INSERT operation must work even in cases where a stimulus is only encountered once, and does not necessarily have any similarity to previously stored items. A random projection scheme, able to immediately encode a new item, can do this.

5.2 Locality sensitive hashing-based approximation

HW-architectures with certain parameter settings (i.e., filter sizes, pooling domains, etc) are equivalent to convolutional neural networks, and with other parameter settings are equivalent to nearest neighbor search algorithms. This correspondence suggests a powerful approximation strategy. It may not be biologically plausible in its details but it is interesting nonetheless. It shares more in common with the random projection strategy than the PCA, thus if it were used by the brain, hippocampus would be the most likely place.

Assume max-pooling for all the following. Locality-sensitive hashing (LSH) is a data structure that supports fast queries by solving an approximate nearest neighbor problem. It can be recast as a data structure for fast querying of HW-modules. Thus it can approximate max-pooling convolutional neural networks analogously to the way it approximates nearest neighbor search.

Many different LSH schemes exist. Inspired by the impressive results of [42], we chose winner-take-all (WTA) hashing [43] for the implementation we used in our experiments.

Figure 1: Examples of faces used in our experiment, here with uniform background. Figure 2: Two-layered HW-architecture, as described in section 7.

6 Approximation Algorithm and Model Architecture

The typical architecture is illustrated in figure 2 and used in section 7. Its consists of two layers of HW-modules: upper layer models cortex (PCA), lower models hippocampus (WTA). The cortical layers used the PCA approximation and the hippocampal layer used the WTA-based approximation described in Algorithm 1.

The data stored in one HW-module is now coupling stored templates with their hashes. An HW-layer is a set of HW-modules. The creation of the candidates set in Line 6 of Algorithm 1 can be done in several ways, with parallelized explicit comparison of the integer-valued hashes for all templates, or via binning and two-stage indexing as in E2LSH [44] when the saving on the larger number of templates compensates the runtime overhead.

1:function INSERT()
3:      Use hashes for amplification
4:end function
5:function QUERY()
6:        using pre-computed See text
7:      Parallelized by GPU, approximates (1)
8:     return
9:end function
Algorithm 1 Insertion and querying of an approximate HW-layer

7 Experiments

We present two experiments illustrating respectively our Ventral Stream and our MTL models. They share most of their architecture, illustrated in figure 2, using the algorithms of section 6. Cortex has two subdivisions. Cortex-1, used for both experiments, models a face-selective region like the fusiform face area (FFA). The S-units of cortex-1 are tuned to images of faces at different angles. There is one HW-module for each familiar individual. Cortex-2, used only for the ventral stream experiment, models a word-selective region like the visual word form area (VWFA). Its S-cells are tuned to images of written names. Within one HW-module, all S-cells are tuned to images of names in different fonts and at different retinal locations. S-cells in the hippocampus are tuned to specific associations in the previous layer. Each hippocampal S-cell is connected to all

C-cells in the cortex: C1 only for ventral stream experiment, and C1 and C2 for MTL experiment. Thus the templates stored by the S-cells are distributed representations over the entire layer below. There is one HW-module in the hippocampus for each individual person = {face, name} to be remembered.

7.1 Ventral Stream experiment: Same-different matching of unfamiliar faces

The ventral stream task was a same-different unfamiliar face matching forced choice (see figure 1), with uniform background, natural background, and occlusions. It is assumed to have no memory demands. In each trial, signatures from the face-tuned cortical component of the architecture (see figure 2

) are computed for both face images in the test pair. The model’s response was taken to be the thresholded cosine similarity between the two signatures. Strong performance on this task required tolerance to 3D rotation in depth. In the training phase, the model was presented with 320 videos (image sequences), each depicting the rotation of a different individual. The test sets were drawn from the images of the remaining 80 individuals. The training phase was taken to be a model of visual development. The individuals of the training phase modeled people with which the subject would be highly familiar: parents, friends, etc.

The interpretation of the training procedure is that high level visual representations are tuned according to a temporal association-based rule. There is evidence from psychophysics [45, 46] and neurophysiology [47, 48] that ventral stream representations are adapted to temporal correlations in this way. Figure 3 shows that our model, HWarch, using the Cortex-1 (FFA-esque) subsystem of figure 2, outperforms two baseline feature representations [49] on the yaw rotation invariant same-different matching of unfamiliar faces (SUFR datasets [49]). Note that both baselines used an SVM (supervised training) whereas our model only compared the cosine similarity between each pair of test faces. Under the proposed cortical (PCA) approximation, the templates of S-cells would correspond to projections of the frames onto principal components. It is assumed that this INSERT operation would run slowly and take many interleaved repetitions of the data (though in the case of our experiment we just computed the PCA). The signature computed by a call to QUERY measures the input’s similarity to the closest frame of each sequence (assuming ). As long as temporally adjacent frames usually depict the same identity face, the signature will remain stable with viewing angle [25, 33]. The level of accuracy achieved is comparable to similar systems [50, 33, 51, 52] that were presented as models of view-tolerant representations in the anterior medial (AM) patch of the macaque face-processing system [4].

Figure 3: Ventral stream task results: our model HWarch, outperforms two baselines from [49]. Figure 4: MTL task results. Error bars: 25- and 75-percentile over 20 repetitions with different development, study, and testing sets.

7.2 MTL experiment: Recall of face-name associations

The MTL experiment tested recall of associations between faces and names. Units in the hippocampal layer (fig. 2) can be regarded as modeling the multimodal cells described by [35]. These cells, discovered in the MTL of human intractable epilepsy patients, respond selectively to visually presented famous faces, e.g., Saddam Hussein’s face, the image of his name, and the sound of it spoken (though we do not address the auditory component here). Critically, [35] also found that some of these multimodal cells were tuned to the researchers themselves. Since the researchers were unknown to the patients prior to the experiment, the multimodal cells tuned to them must have quickly acquired this selectivity.

This experiment explores multimodal binding. Units in the top layer of figure 2 can be regarded as modeling the MTL cells recorded by [35]. The experiment was divided into three phases. Development phase: In first phase, Cortex-1 (the face area) was trained in the exact same way as in experiment 1. Cortex-2 was trained analogously, but in this case, each template book contained a set of template images depicting equivalent words (4-letter names) at a range of positions and fonts. Both cortex-1 and 2 used the PCA approximation. Study phase: The training of the hippocampal layer modeled the task of meeting a set of people at an event, say a poster session. That is, each individual consists of a face and a nametag. The two pixel representations are concatenated and encoded in Cortex-1/2. However, the model never sees the nametag or face from all views and the names vary both in retinal location and typeface. Training consisted of INSERT-ing all the images associated with the same individual to an (initially empty) hippocampal HW-module representing the episodic memory of that individual. Since it is assumed that there is not enough time for there to be an effect on cortex, none of the faces or names used to train the cortical layers were used in the study phase. Test phase: A test of recalling names from faces, or faces from names. In both cases, the goal is for the maximally activated hippocampal HW-module to be the one containing the representation of the query.

Figure 4 shows results from a simulation of the episodic recall of names from taxes task, using the full architecture of figure 2. The neocortex parameter that we vary is the number of faces used for the cortical model in the first layer. The hippocampal parameters were the WTA hashing parameters: he WTA’s metric-determining parameter , the number of hashes appearing in Alg. 1, and the number of -bit integers used per hash , see [42].

The assumption underlying this simulation is that hippocampus learns associations between arbitrary items. This simulation tests both the usual case in the memory literature when the probe stimulus (i.e., the query) is exactly one of the items encountered during the study phase, and the less commonly studied case where a correct probe stimulus is a semantically equivalent item. We argue that the latter is the more ecologically valid of the two tasks. There are two mechanisms through which recall decreases with increasing study set size: First, the cortical representation “compresses” differences in the input, making them appear more similar. This is useful for generalizing rotation invariance, but problematic for the recall task. Second, the LSH approximation can miss some candidate items when there are too many. The former models “capacity limits” due to the cortical afferents while the latter models capacity limits arising from the hippocampus itself.

8 Discussion

There are myriad potential gains from thinking about perception and memory in an integrated way. We’ve already pointed out the additional useful constraints on MTL modeling posed by realistic perceptual inputs, but the benefits of an integrated view are not unidirectional. For example, insight into outstanding questions in vision like the learning of invariant representations may be informed by a modern view of semantic memory, replay and consolidation. The primary role of sensory cortex can be thought of as learning the statistics of the input distribution. However, a learning organism in the real world may not wish to merely be a slave to the statistics of its environment but may want to bias its learning towards inputs that have some relevance to its goals. Hippocampal replay may provide just such a mechanism by preferentially replaying rewarding episodes during sleep and thereby presenting neocortex with more examples from which to learn [53]. In this way the statistics of the environment can be circumvented.

9 Supplementary Information

9.1 Recognition experiment details

We tested three different datasets from the Subtasks of Unconstrained Face Recognition (SUFR) benchmark collection

[49]. They were 1. yaw rotation with uniform background (shown in Figure 1 of main text)  2. yaw rotation with natural image background, and 3. yaw rotation with occlusion (demanding invariance to the presence or absence of sunglasses). There were 400 individual faces and 19 images of each face (19 different orientations) in task 1 and 2. Task 3 had images per individual. The classifier was the thresholded cosine similarity between the two images to be compared. The threshold was optimized on the training set.

9.2 Unsupervised versus error-based learning

We focus on associations that can be learned without explicit error signals. This likely constitutes the majority of the ventral stream’s learning [54], but only a part of the MTL’s.

9.3 Feedforward models

One potential caveat to our proposal is that it neglects generative phenomena and top-down influences. The framework of this article extends feedforward models of the ventral stream into the medial temporal lobe. Feedforward models may account for neurophysiology results obtained by analyses restricted to a window around the time the first volley of spikes arrives in a region or behavioral effects elicited from brief presentations ( 100ms) and masking [55, 10, 56, 16, 57]. They are not expected to capture attentional effects, priming / adaptation effects, mental rotation, and a host of other ventral stream-related phenomena.

When two regions are known to be monosynaptically connected, a feedforward model is the most straightforward hypothesis. Demonstrating a feedforward solution to a difficult computational problem motivates experiments under the aforementioned brief presentation conditions and detailed examination of neuronal response latencies. Note however, it does not constitute a denial of other phenomena.

9.4 Neuronal selectivity latencies

In accord with our proposal, selectivity latencies increase from the anterior ventral stream (100-200ms) [3, 56] to MTL (200-500ms) [58, 59]. It is sometimes claimed that the significantly longer latencies in MTL333though MTL latencies may be shorter in macaque data [60] support the hypothesis that “long loop” recurrence is necessary to generate selectivity in these regions e.g. [61]. Without rejecting that hypothesis, we note that such results are also compatible with a feedforward model. One explanation is that additional integration time per stage is needed to integrate cross-modal inputs, especially if different information processing streams undergo differing numbers of stages prior to MTL. Another possibility is that each MTL area actually contains several processing stages. In this vein, it’s notable that MTL areas are typically defined by cytoarchitectural and anatomical criteria rather than physiological criteria as visual areas often are (containing a map of visual space).

10 Acknowledgments

The authors would like to thank Peter Dayan, Guillaume Desjardins, Greg Wayne, Andrei Rusu, Vlad Mnih, Dharshan Kumaran, and Fernando Pereira for helpful comments on early versions of this manuscript, Ioannis Antonoglou for engineering, and Adam Cain for graphic design.

plus 0.3ex


  • [1] H. Bülthoff, S. Edelman, Proceedings of the National Academy of Sciences 89, 60 (1992).
  • [2] M. J. Tarr, H. H. Bülthoff, Journal of Experimental Psychology: Human Perception and Performance 21, 1494 (1995).
  • [3] C. P. Hung, G. Kreiman, T. Poggio, J. J. DiCarlo, Science 310, 863 (2005).
  • [4] W. A. Freiwald, D. Tsao, Science 330, 845 (2010).
  • [5] E. Tulving, Elements of episodic memory (Oxford University Press, 1985).
  • [6] P. Andersen, R. Morris, D. Amaral, T. Bliss, J. O’Keefe, The hippocampus book (Oxford University Press, 2006).
  • [7] D. Hubel, T. Wiesel, The Journal of Physiology 160, 106 (1962).
  • [8] J. L. McClelland, B. L. McNaughton, R. C. O’Reilly, Psychological review 102, 419 (1995).
  • [9] K. A. Norman, R. C. O’Reilly, Psychological review 110, 611 (2003).
  • [10] T. Serre, A. Oliva, T. Poggio, Proceedings of the National Academy of Sciences of the United States of America 104, 6424 (2007).
  • [11] W. A. Suzuki, D. G. Amaral, Journal of Comparative Neurology 350, 497 (1994).
  • [12] S. Kali, P. Dayan, Nature Neuroscience 7, 286 (2004).
  • [13] M. Riesenhuber, T. Poggio, Nature Neuroscience 2, 1019 (1999).
  • [14] T. S. Lee, D. B. Mumford, Journal of the Optical Society of America 20, 1434 (2003).
  • [15] E. Rolls, Frontiers in Computational Neuroscience 6 (2012).
  • [16] J. J. DiCarlo, D. Zoccolan, N. C. Rust, Neuron 73, 415 (2012).
  • [17] N. Sadato, et al., Nature 380, 526 (1996).
  • [18] B. Röder, O. Stock, S. Bien, H. Neville, F. Rösler, European Journal of Neuroscience 16, 930 (2002).
  • [19] M. Sur, P. Garraghty, A. Roe, Science 242, 1437 (1988).
  • [20] Y. LeCun, et al., Neural computation 1, 541 (1989).
  • [21] K. Fukushima, Biological Cybernetics 36, 193 (1980).
  • [22] A. Krizhevsky, I. Sutskever, G. Hinton, Advances in neural information processing systems (Lake Tahoe, CA, 2012), vol. 25, pp. 1106–1114.
  • [23] O. Abdel-Hamid, A. Mohamed, H. Jiang, G. Penn, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2012), pp. 4277–4280.
  • [24] D. Marr, Vision: A computational investigation into the human representation and processing of visual information (Henry Holt and Co., Inc., New York, NY, 1982).
  • [25] F. Anselmi, et al., arXiv:1311.4158v3 [cs.CV] (2013).
  • [26] Y. LeCun, Y. Bengio, The handbook of brain theory and neural networks pp. 255–258 (1995).
  • [27] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber, T. Poggio, IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 411 (2007).
  • [28] E. Fix, J. L. Hodges, Defense Technical Information Center (DTIC) report ADA800276 (1951).
  • [29] A. Coates, A. Karpathy, A. Y. Ng, Advances in Neural Information Processing Systems (2012), pp. 2681–2689.
  • [30] L. Isik, J. Z. Leibo, T. Poggio, Front. Comput. Neurosci. 6 (2012).
  • [31] Q. Liao, J. Z. Leibo, T. Poggio, arXiv preprint arxiv:1409.3879 (2014).
  • [32] P. Földiák, Neural Computation 3, 194 (1991).
  • [33] Q. Liao, J. Z. Leibo, T. Poggio, Advances in Neural Information Processing Systems (NIPS) (Lake Tahoe, CA, 2013).
  • [34] H. Eichenbaum, Memory, amnesia, and the hippocampal system (MIT press, 1993).
  • [35] R. Q. Quiroga, A. Kraskov, C. Koch, I. Fried, Current Biology 19, 1308 (2009).
  • [36] M. Riesenhuber, T. Poggio, Current Opinion in Neurobiology 12, 162 (2002).
  • [37] E. Oja, Journal of mathematical biology 15, 267 (1982).
  • [38] T. Sanger, Neural networks 2, 459 (1989).
  • [39] E. Oja, Neural Networks 5, 927 (1992).
  • [40] W. B. Johnson, J. Lindenstrauss, Contemporary mathematics 26, 189 (1984).
  • [41] L. Wiskott, T. Sejnowski, Neural computation 14, 715 (2002).
  • [42] T. Dean, et al.,

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    (IEEE, Portland, OR, USA, 2013), pp. 1814–1821.
  • [43] J. Yagnik, D. Strelow, D. A. Ross, R.-s. Lin, IEEE International Conference on Computer Vision (ICCV) (IEEE, Barcelona, Spain, 2011), pp. 2431–2438.
  • [44] A. Andoni, P. Indyk, 47th Annual IEEE Symposium on Foundations of Computer Science (2006), pp. 459–468.
  • [45] G. Wallis, H. H. Bülthoff, Proceedings of the National Academy of Sciences of the United States of America 98, 4800 (2001).
  • [46] D. Cox, P. Meier, N. Oertelt, J. J. DiCarlo, Nature Neuroscience 8, 1145 (2005).
  • [47] N. Li, J. J. DiCarlo, Science 321, 1502 (2008).
  • [48] N. Li, J. J. DiCarlo, Neuron 67, 1062 (2010).
  • [49] J. Z. Leibo, Q. Liao, T. Poggio, International Joint Conference on Computer Vision, Imaging and Computer Graphics, VISAPP (SCITEPRESS, Lisbon, Portugal, 2014).
  • [50] J. Z. Leibo, J. Mutch, T. Poggio, Advances in Neural Information Processing Systems (NIPS) (Granada, Spain, 2011).
  • [51] Q. Liao, J. Z. Leibo, Y. Mroueh, T. Poggio, arXiv preprint arXiv:1311.4082 (2014).
  • [52] J. Z. Leibo, Q. Liao, F. Anselmi, T. Poggio, The invariance hypothesis implies domain-specific regions in visual cortex, Tech. rep., CBMM (2014).
  • [53] A. C. Singer, L. M. Frank, Neuron 64, 910 (2009).
  • [54] N. Li, J. J. DiCarlo, The Journal of Neuroscience 32, 6611–6620 (2012).
  • [55] S. Thorpe, D. Fize, C. Marlot, Nature 381, 520 (1996).
  • [56] H. Liu, Y. Agam, J. R. Madsen, G. Kreiman, Neuron 62, 281 (2009).
  • [57] L. Isik, E. M. Meyers, J. Z. Leibo, T. Poggio, Journal of Neurophysiology 111, 91 (2014).
  • [58] G. Kreiman, C. Koch, I. Fried, Nature neuroscience 3, 946 (2000).
  • [59] F. Mormann, et al., The Journal of Neuroscience 28, 8865 (2008).
  • [60] W. A. Suzuki, E. K. Miller, R. Desimone, Journal of neurophysiology 78, 1062 (1997).
  • [61] D. Kumaran, J. L. McClelland, Psychological review 119, 573 (2012).