Natural behavior often involves a continuous series of related images, often while the subject is immobile. How is this information processed across the cortico-hippocampal circuit? The hippocampus is crucial for episodic memory(1)–(3), but most rodent single unit studies require spatial exploration(4)–(6) or active engagement(7). Hence, we investigated neural responses to a silent, iso-luminant, black and white movie in head-fixed mice without any task or locomotion demands, or rewards. The activity of most neurons (97%, 6554/6785) in the thalamo-cortical visual areas was significantly modulated by the 30s long movie clip. Surprisingly, a third (33%, 3379/10263) of hippocampal –dentate gyrus, CA1 and subiculum– neurons showed movie-selectivity, with elevated firing in specific movie sub-segments, termed movie-fields. On average, a cell had more than 5 movie-fields in visual areas, but only 2 in hippocampal areas. The movie-field durations in all brain regions spanned an unprecedented 1000-fold range: from 0.02s to 20s, termed mega-scale coding. Yet, the total duration of all the movie-fields of a cell was comparable across neurons and brain regions, partly due to broader movie-fields in hippocampal areas, indicating greater sequence coding. Consistently presentation of the movie images in a scrambled sequence virtually abolished hippocampal but not visual-cortical selectivity. The enhancement of sequential movie tuning compared to the scrambled sequence was eight-fold greater in hippocampal than visual areas, further supporting visual sequence encoding. Thus, a movie was encoded in all mouse-brain areas investigated. Similar results are likely to hold in primates and humans. Hence, movies could provide a unified way to probe neural mechanisms of non-spatial information processing and memory across brain regions and species.
This manuscript analyzes large-scale Neuropixels recordings from visual areas and hippocampus of mice passively viewing repeated clips of a movie and reports that neurons respond with elevated firing activities to specific, continuous sequences of movie frames. The important results support a role of rodent hippocampal neurons in general episode encoding and advance understanding of visual information processing across different brain regions. The strength of evidence for the primary conclusion is solid, but some technical limitations of the study were identified that merit further analyses.
In addition to the position and orientation of simple visual cues, like Gabor patches and drifting gratings(8), primary visual cortical responses are also direction selective(9), and show predictive coding(10), suggesting that the temporal sequence of visual cues influences neural firing. Accordingly, these and higher visual cortical neurons also encode a sequence of visual images, i.e., a movie(11)–(13). The hippocampus is farthest downstream from the retina in the visual circuit. The rodent hippocampal place cells encode spatial or temporal sequences(2),(14)–(21) and episode-like responses(22)–(24). However, these responses typically require active locomotion(25), and they are thought to be non-sensory responses(26). Primate and human hippocampal responses are selective to specific sets of visual cues, e.g., the object-place association(27), their short-term(1) and long-term(28) memories, cognitive boundaries between episodic movies(29), and event integration for narrative association(30). However, despite strong evidence for hippocampal role in episodic memory, the hippocampal encoding of a continuous sequence of images, i.e., a visual episode, is unknown.
We used a publicly available dataset (Allen Brain Observatory – Neuropixels Visual Coding, © 2019 Allen Institute). Mice were monocularly shown a 30s clip of a continuous segment from the movie Touch of Evil (Welles, 1958)(31) (Extended Data Fig. 1). Mice were head-fixed but were free to run on a circular disk. A total of 17048 broad spiking, active, putatively excitatory neurons were analyzed, recorded using 4-6 Neuropixel probes in 24 sessions from 24 mice (See Methods).
The majority of neurons in the visual areas (Lateral geniculate nucleus LGN, primary visual cortex V1, higher visual areas: antero-medial and medio-lateral AM-PM) were modulated by the movie, consistent with previous reports (Extended Data Fig. 2)(11)–(13),(32). Surprisingly, neurons from all parts of the hippocampus (dentate gyrus DG, CA3, CA1, subiculum SUB) were also clearly modulated (Fig. 1), with reliable, elevated spiking across many trials in small movie segments. To quantify selectivity in an identical, firing rate- and threshold-independent fashion across brain regions, we computed the z-scored sparsity(33),(34) of neural selectivity (See Methods). Cells with z-scored sparsity >2 were considered significantly (p<0.03) modulated. The areas V1 (97.3%) and AM-PM (97.1%) had the largest percentage of movie tuned cells, much higher than reported earlier(11) (∼40%), perhaps because we analyzed extracellular spikes, while the previous study used calcium imaging. The majority of neurons in LGN (89.2%) too showed significant modulation by the movie. Thus, the vast majority of thalamo-cortical neurons were significantly modulated by the movie.
Movie selectivity was prevalent in the hippocampal regions too, despite head fixation, dissociation between self-movements and visual cues as well as the absence of rewards, task, or memory demands (Fig. 1a-d). Subiculum, the output region of the hippocampus, farthest removed from the retina, had the largest fraction (44.6% Fig. 1d) of movie-tuned neurons, followed by the upstream CA1 (33.6%, Fig. 1c) and dentate gyrus (33.1%, Fig. 1a). However, CA3 movie selectivity was nearly half as much (17.3%, Fig. 1b). This is unlike place cells, where CA3 and CA1 selectivity are comparable(35),(36) and subiculum selectivity is weaker(37).
To confirm these findings, we did several controls. Running alters neural activity in visual areas(38),(39) and hippocampus(40),(41). Hence, we used the data from only the stationary epochs (see Methods) and only from sessions with at least 300 seconds of stationary data (17 sessions, 24906 cells). Movie tuning was unchanged in this data (Extended Data Fig. 3). This is unlike place cells where spatial selectivity is greatly reduced during immobility(5),(6). Neurons recorded simultaneously from the same brain region also showed different selectivity patterns (Extended Data Fig. 4). Thus, nonspecific effects such as running cannot explain the brain wide movie selectivity.
Hippocampal neurons have one or two place fields in typical mazes which take a few seconds to traverse(42). In larger arenas that take tens of seconds to traverse, the number of peaks per cell and the peak duration increases(43)–(46). Peak detection for movie tuning is nontrivial because neurons have nonzero background firing rate, and the elevated rates cover a wide range (Fig. 1). We developed a novel algorithm to address this (see Methods). On average, V1 neurons had the largest number of movie-fields (Fig. 2a, mean±s.e.m.=10.4±0.1), followed by LGN (8.6±0.3) and AM-PM (6.3±0.07). Hippocampal areas had significantly fewer movie-fields per cell: dentate gyrus (2.1±0.1), CA3 (2.8±0.3), CA1(2.0±0.02) and subiculum (2.1±0.05). Thus, the number of movie-fields per cell was smaller than the number of place-fields per cell in comparably long spatial tracks(43)–(48), but a handful of hippocampal cells had more than 5 movie-fields (Extended Data Fig. 5).
Typical receptive field size increases as one moves away from the retina in the visual hierarchy(31). A similar effect was seen for movie-field durations. On average, hippocampal movie-fields were longer than visual regions (Fig. 2b). But there were many exceptions –movie-fields of LGN (median±s.e.m., here and subsequently, unless stated otherwise, 308.5±33.9 ms) were twice as long as in V1 (156.6±9.2ms). Movie-fields of subiculum (3169.9±169.8 ms) were significantly longer than CA1 (2786.1±77.5 ms) and nearly three-fold longer than the upstream CA3 (979.1±241.1 ms). However, the dentate movie-fields (2113.2±172.4 ms) were two-fold longer than the downstream CA3. This is similar to the patterns reported for CA3, CA1 and dentate gyrus place cells(48). But others have claimed that CA3 place fields are slightly bigger than CA1(49), whereas movie-fields showed the opposite pattern.
The movie-field durations spanned a 500-1000-fold range in every brain region investigated (Fig. 2e). This mega-scale scale is unprecedentedly large, nearly 2 orders of magnitude greater than previous reports in place cells(43),(45). Even individual neurons showed 100-fold mega-scale responses (Fig. 2c & d) compared to less than 10-fold scale within single place cells(43),(45). The mega-scale tuning within a neuron was largest in V1 and smallest in subiculum (Fig. 2e). This is partly because the short duration movie-fields in hippocampal regions were typically neither as narrow and nor as prominent as in the visual areas (Extended Data Fig. 6).
Despite these differences in mega-scale tuning across different brain areas, the total duration of elevated activity, i.e., the sum of movie-field durations within a single cell, was remarkably conserved across neurons within and across brain regions (Fig. 2f). Unlike movie-field durations, which differed by more than ten-fold between hippocampal and visual regions, cumulative durations were quite comparable, ranging from 6.2s (V1) to 10.2s (CA3) (Fig. 2f, LGN=8.8±0.21sec, V1=6.2±0.09, AM-PM=7.8±0.09, DG=9.4±0.26, CA3=10.2±0.46, CA1=9.1±0.12, SUB=9.5±0.27). Thus, hippocampal movie-fields are longer and less multi-peaked than visual areas, such that the total duration of elevated activity was similar across all areas, spanning about a fourth of the movie, comparable to the fraction of large environments in which place cells are active(45),(47),(48). To quantify the net activity in the movie-fields, we computed the total firing in the movie-fields (i.e., the area under the curve for the duration of the movie-fields), normalized by the expected discharge from the shuffled response. Net movie-field discharge was also comparable across brain areas, but maximal in V1 and least in subiculum (Fig. 2g).
Many movie-fields showed elevated activity spanning up to several seconds, suggesting rate-code like encoding (Fig. 2h). However, some cells showed movie-fields with elevated spiking restricted to less than 50ms, similar to responses to briefly flashed stimuli in anesthetized cats(12),(13),(50). This is suggestive of a temporal code, characterized by low spike timing jitter(51). Such short-duration movie-fields were not only common in the thalamus, but also AM-PM, three synapses away from the retina. A small fraction of cells in the hippocampal areas, more than five synapses away from the retina, too showed such temporally coded fields (Fig. 2h).
To determine the stability and temporal-continuity of movie tuning across the neural ensembles we computed the population vector overlap between even and odd trials(52) (see Methods). Population response stability was significantly greater for tuned than for untuned neurons (Extended Data Fig. 7). The population vector overlap around the diagonal was broader in hippocampal regions than visual cortical and LGN, indicating longer temporal-continuity, reflective of their longer movie-fields. Further, the population vector overlap away from the diagonal was longer around frames 400-800 in all brain areas due to the longer movie-fields in that movie segment (see below).
Are all movie frames represented equally by all brain areas? The duration and density of movie-fields varied as a function of the movie frame and brain region (Extended Data Fig. 8). We hypothesized that this variation could correspond to the change in visual content from one frame to the next. Hence for comparison, we quantified the similarity between adjacent movie frames as the correlation coefficient between corresponding pixels and termed it as frame-to-frame (F2F) correlation. The majority of brain regions had substantially reduced density of movie-fields between the movie frames 400 to 800, but the movie-fields were longer in this region. This effect was greater in the visual areas than hippocampal. Using significantly tuned neurons, we computed the average neural activity in each brain region at each point in the movie (see Methods). Although movie-fields (Fig. 3a), or just the tallest movie-field per cell (Fig. 3b), covered the entire movie, the peak normalized, ensemble activity level of all brain regions showed significant overrepresentation, i.e., deviation from the norm, in certain parts of the movie (Fig. 3c, see Methods). This was most pronounced in V1 and higher visual areas AM-PM. The number of movie frames with elevated ensemble activity was higher in visual areas than hippocampal regions (Fig. 3d), and also this modulation (see Methods) was smaller in hippocampus and LGN, compared to visual cortical regions (Fig. 3e).
Using the significantly tuned neurons, we also computed the average neural activity in each brain region corresponding to each frame in the movie, without peak rate normalization (see Methods). The degree of continuity between the movie frames, quantified as above (F2F correlation), was inversely correlated with the ensemble rate modulation in all areas except DG, CA3 and CA1 (Fig. 3f and g). As expected for a continuous movie, this F2F correlation was close to unity for most frames, but highest in the latter part of the movie where the images changed more slowly. The population wide elevated firing rates, as well as the smallest movie-fields, occurred during the earlier parts (Extended Data Fig. 8). Thus, the movie-code was stronger in the segments with greatest change across movie frames. These results show differential ensemble-wide representation of the movie across brain regions.
If these responses were purely visual, a movie made of scrambled sequence of images would generate equally strong or even stronger selectivity due to the even larger change across movie frame, despite the absence of similarity between adjacent frames. To explore this possibility, we investigated neural selectivity when the same movie frames were presented in a fixed but scrambled sequence (scrambled movie). The within frame and the total visual content was identical between the continuous and scrambled movies, and the same sequence of images was repeated many times in both experiments (see Methods). But there was no correlation between adjacent frames, i.e., visual continuity, in the latter (Fig.4a).
For all brain regions investigated, the continuous movie generated significantly greater modulation of neural activity than the scrambled sequence (Fig. 4b). Middle 20 trials of the continuous movie were chosen as the appropriate subset for comparison since they were chronologically closest to the scrambled movie presentation. This preference for sequential over scrambled movie was the greatest in hippocampal regions where the percentage of significantly tuned neurons (4.4%, near chance level of 2.3%) reduced more than 4-fold compared to the continuous movie (17.8%, after accounting for the lesser number of trials, see Methods). This was unlike visual areas where the scrambled (80.4%) and the continuous movie (92.4%) generated similar prevalence levels of selectivity (Fig. 4b). The few hippocampal cells which had significant selectivity to the scrambled sequence, did not have long-duration responses, but only very short, ∼50ms long responses (Fig. 4d), reminiscent of, but even sharper than human hippocampal responses to flashed images(28). To estimate the effect of continuous movie compared to the scrambled sequence on individual cells, we computed the normalized difference between the continuous and scrambled movie selectivity for cells which were selective in either condition (Fig. 4c, see Methods). This visual continuity index was more than eight-fold higher in hippocampal areas (87.8%) compared to the visual areas (10.6%).
The pattern of increasing visual continuity index as we moved up the visual hierarchy, largely paralleled the anatomic organization(53), with the greatest sensitivity to visual continuity in the hippocampal output regions, CA1 and subiculum, but there were notable exceptions. The primary visual cortical neurons showed the least reduction in selectivity due to the loss of temporally contiguous content, whereas LGN neurons, the primary source of input to the visual cortex and closer to the periphery, showed far greater sensitivity (Fig. 4c).
Many visual cortical neurons were significantly modulated by the scrambled sequence, but their number of movie-fields per cell was greater and their duration was shorter than during the continuous movie (Extended Data Fig. 9 and 10). This could occur due to the loss of frame-to- frame correlation in the scrambled sequence. The average activity of the neural population in V1 and AM-PM showed significant deviation even with the scrambled movie, comparable to the continuous movie, but this multi-unit ensemble response was uncorrelated with the frame-to- frame correlation in the scrambled sequence (Extended Data Fig. 11). A substantial fraction of visual cortical and LGN responses to the scrambled sequence could be rearranged to resemble continuous movie responses (Extended Data Fig. 12, see Methods). The latency needed to shift the responses was least in LGN and largest in AM-PM, as expected from the feed-forward anatomy and the model of visual information processing (Extended Data Fig. 12). This rearrangement did not recreate the continuous movie responses above chance levels in the hippocampal regions (example cells in Fig. 4e, also see Extended Data Fig. 12 for statistics and details).
Population vector decoding of the ensemble of a few hundred place cells is sufficient to decode the rat’s position using place cells(54), and the position of a passively moving object(55). Using similar methods, we decoded the movie frame number (see Methods). Continuous movie decoding was better than chance in all brain regions analyzed (Fig. 4f). Scrambled movie decoding was significantly weaker yet above chance level (as expected from shuffles, see Methods) in visual areas, but not in CA3 and dentate gyrus. But CA1 and subiculum neuronal ensembles could be used to decode scrambled movie frame number slightly above chance levels (Fig. 4g). Similarly, the population overlap between even and odd trials for scrambled sequence was strong for visual areas, and weaker in hippocampal regions, but significantly greater than untuned neurons in hippocampal regions (Extended Data Fig. 13).
To understand how neurons encode a continuously unfolding visual episode, we investigated the neural responses in the rodent brain to an isoluminant human movie. As expected, neural activity in all thalamo-cortical visual areas was significantly modulated, with elevated activity in response to specific visual content. Most (96.6%, 6554/6785) of thalamo-cortical neurons showed significant movie tuning, nearly double that reported for the classic stimuli such as Gabor patches in the same dataset(31). Remarkably, a third of hippocampal neurons (32.9%, 3379/10263) were also movie-tuned, comparable to the fraction of neurons with significant spatial selectivity in mice(56) and bats(57). While the hippocampus is implicated in episodic memory, rodent hippocampal responses are largely studied in the context of spatial maps or place cells, and more recently in other tasks which requires active locomotion or active engagement(7),(58).
However, unlike place cells(5),(6), movie-tuning remained intact during immobility in all brain areas studied. This dissociation of the effect of mobility on spatial and movie selectivity agrees with the recent reports of dissociated mechanisms of episodic encoding and spatial navigation in human amnesia(59).
Across all brain regions, neurons showed a mega-scale encoding by movie-fields varying in duration by up to 1000-fold, similar to, but far greater than recent reports of 10-fold multi-scale responses in the hippocampus(43)–(48),(60). Importantly, mega-scale responses were found within single neurons as well, which can enable encoding of both finer and broader details of an episode in the same neuron. This could provide greatly enhanced episodic memory and recall.
The response latency was highest in AM-PM, then V1 and least in LGN, thus following the visual hierarchy. Among the visual areas, continuous movie tuning was greater in V1, more than the upstream LGN. This is surprising but can explain the recent findings that silencing V1 reduces movie tuning in the thalamus(61). Similarly, the preference for the continuous movie over scrambled sequence was greater in LGN and AM-PM than V1, which does not follow the expected hierarchy of visual processing. This effect was even greater in the hippocampal areas than any visual regions. Temporal integration window(62)–(64) as well as intrinsic timescale of firing(31) increases along the anatomical hierarchy in the cortex, with the hippocampus being farthest removed from the retina(53). This hierarchical organization could explain the longer movie-fields and several fold greater preference for the continuous movie over scrambled sequence in the hippocampus. But, unlike reports of image-association memory in the inferior temporal cortex for unrelated images(65), only a handful hippocampal neurons showed selective responses to the scrambled sequence. These results, along with the longer duration of hippocampal movie-fields could mediate visual-chunking or binding of a sequence of events.
Could the brain-wide mega-scale tuning be an artifact of poor unit isolation, e.g., due to an erroneous mixing of two neurons, one with very short and another with very long movie-fields? This is unlikely since the LGN and visual cortical neural selectivity to classic stimuli (Gabor patches, drifting gratings etc.) in the same dataset was similar to that reported in most studies(31) whereas poor unit isolation should reduce these selective responses. However, to directly test this possibility, we calculated the correlation between the unit isolation index (or fraction of refractory violations) and the mega-scale index of the cell, while factoring out the contribution of mean firing rate (Extended Data Fig. 14). This correlation was not significant (p>0.05) for any brain areas.
Do the movie fields arise from the same mechanism as place fields? Studies have shown that when rodents are passively moved along a linear track that they had explored(6), or when the images of the environment around a linear track was played back to them(5), some hippocampal neurons generated spatially selective activity. Since the movie clip involved change of spatial view, one could hypothesize that the movie fields are just place fields generated by passive viewing. This is unlikely for several reasons. Mega-scale movie fields were found in all brain regions investigated, including the lateral geniculate nucleus and the primary visual cortex. There is no evidence of place cells in these brain regions. Further, in prior passive viewing experiments, the rodents were shown the same narrow linear track, like a tunnel, that they had previously explored actively to get food rewards at specific places. In contrast, in current experiments, these mice had never actively explored the space shown in the movie, nor obtained any rewards. Active exploration of a maze, combined with spatially localized rewards engages multisensory mechanisms resulting in increased place cell activation(17),(23),(66) which are entirely missing in these experiments during passive viewing of a movie, presented monocularly, without any other multisensory stimuli and without any rewards. Compared to the exploration of a real-world maze, exploration of a visually identical virtual world causes 60% reduction in CA1 place cell activation(67). In contrast, there was no evidence of significant CA1 neural shutdown during movie viewing, compared to the blank screen presentation (Extended Data Fig. 14).
A recent study showed that CA1 neurons encode the distance, angle, and movement direction of motion of a vertical bar of light(55), consistent with the position of hippocampus in the visual circuitry(53). Do those findings predict the movie tuning herein? There are indeed some similarities between the two experimental protocols –purely passive optical motion without any self-motion or rewards. However, there are significant differences too; similar to place cells in the real and virtual worlds(34), all the cells tuned to the moving bar of light had single receptive fields with elevated responses lasting a few seconds; there were neither punctate responses nor even 10-fold variation in neural field durations, let alone the 1000-fold change reported here. Finally, those results were reported only in area CA1, while the results presented here cover nearly all the major stations of the visual hierarchy.
Notably, hippocampal neurons did not encode Gabor patches or drifting gratings in the same dataset, indicating the importance of temporally continuous sequences of images for hippocampal activation(31). This is consistent with the hypothesis that the hippocampus is involved in coding spatial sequences(16),(24),(68). However, unlike place cells that degrade in immobile rats, hippocampal movie tuning was unchanged in the immobile mouse. Further, the scrambled sequence too was presented in the same sequence many times, yet movie tuning dropped to chance level in the hippocampal areas. Unlike visual areas, scrambled sequence response of hippocampal neurons could not be rearranged to obtain the continuous movie response. This shows the importance of continuous, episodic content instead of mere sequential recurrence of unrelated content for rodent hippocampal activation.
These results complement recent findings of spatial modulation of visual cortical neurons and coordinated activity of visual and hippocampal neurons during navigation(69),(70). Our findings open up the possibility of studying thalamic, cortical, and hippocampal brain regions in a simple, passive, and purely visual experimental paradigm and extend comparable convolutional neural networks(11) to have the hippocampus at the apex(53). Further, our results here bridge the long-standing gap between the hippocampal rodent and human studies(29),(71)–(73), where natural movies can be decoded from fMRI signals in immobile humans(74). This brain-wide mega-scale encoding of a human movie episode and enhanced preference for visual continuity in the hippocampus compared to visual areas supports the hypothesis that the rodent hippocampus could play a role in non-spatial episodic memories, consistent with classic findings in humans(1) and agrees with a more generalized, representational framework(75),(76) of episodic memory where it encodes temporal patterns. Similar responses are likely across different species, including primates. Thus movie-coding can provide a unified platform to investigate the neural mechanisms of episodic coding, learning and memory.
We used the Allen Brain Observatory – Neuropixels Visual Coding dataset (© 2019 Allen Institute, https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels). This website and related publication(31) contain detailed experimental protocol, neural recording techniques, spike sorting etc. Briefly, and of relevance here, prior to implantation with Neuropixel probes, mice passively viewed the entire range of images including drifting gratings, Gabor patches and the movies of interest here. Videos of the body and eye movements were obtained at 30Hz and synced to the neural data and stimulus presentation using a photodiode. Movies were presented monocularly on an LCD monitor with a refresh rate of 60Hz, positioned 15cm away from the mouse’s right eye and spanned 120°x95°. 30 trials of the continuous movie presentation were followed by 10 trials of the scrambled movie. Next was a presentation of drifting gratings, followed by a quiet period of 30 minutes where the screen was blank. Then the second block of drifting gratings, scrambled movie and continuous movie was presented.
Neural spiking data was sampled at 30 kHz with a 500Hz high pass filter. Spike sorting was automated using Kilosort2(77). Output of Kilosort2 was post-processed to remove noise units, characterized by unphysiological waveforms. Neuropixel probes were registered to a common co-ordinate framework(78). Each recorded unit was assigned to a recording channel corresponding to the maximum spike amplitude and then to the corresponding brain region. Broad spiking units identified as those with average spike waveform duration (peak to trough) between 0.45 to 1.5ms were analyzed herein.
Movie tuning quantification
The movie consisted of 900 frames: 30s total, 30Hz refresh rate, 33.3ms per frame. At the first level of analysis, spike data were split into 900 bins, each 33.3ms wide (the bin size was later varied systematically to detect mega-scale tuning, see below). The resulting tuning curves were smoothed with a Gaussian window of σ=66.6 ms or 2 frames. The degree of modulation and its significance was estimated by the sparsity s as below, and as previously described methods(55).
where rn is the firing rate in the nth frame or bin and N=900 is the total number of bins. Statistical significance of sparsity was computed using a bootstrapping procedure, which does not assume a normal distribution. Briefly, for each cell, the spike train as a function of the frame number from each trial were circularly shifted by different amounts and the sparsity of the randomized data computed. This procedure was repeated 100 times with different amounts of random shifts. The mean value and standard deviation of the sparsity of randomized data were used to compute the z-scored sparsity of observed data using the function zscore in MATLAB. The observed sparsity was considered statistically significant if the z-scored sparsity of the observed spike train was greater 2, which corresponds to p<0.023 in a one tailed t-test. Similar method was used to quantify significance of the scrambled movie tuning, as well as for the subset of data with only stationary epochs, or its equivalent subsample (see below). Middle 20 trials of the continuous movie were used in comparisons with the scrambled movie in Fig. 4, to ensure a fair comparison by using same number of trials, with similar time delays across measurements.
Stationary epoch identification
To eliminate the confounding effects of changes in behavioral state associated with running, we repeated our analysis in stationary epochs, defined as epochs when the running speed remained less than 2cm/sec for this period, as well as at least 5 seconds before and after this period. Analysis was further restricted to sessions with at least 5 total minutes of these epochs during the 60 trials of continuous movie presentation. To account for using lesser data of the stationary epochs, we compared the tuning using a random subsample of data, regardless of running or stopping and compared the two results for difference in selectivity.
Mega-scale movie-field detection in tuned neurons
For neurons with significant movie-sparsity, i.e., movie-tuned, the movie response was first recalculated at a higher resolution of 3.33ms (10 times the frame rate of 33.3ms). The findpeaks function in MATLAB was used to obtain peaks with prominence larger than 110% (1.1x) the range of firing variation obtained by chance, as determined from a sample shuffled response. This calculation was repeated at different smoothing values (logarithmically spaced in 10 Gaussian smoothing schemes with σ ranging from 6.7ms to 3430ms), to ensure that long as well as short movie-fields were reliably detected and treated equally. For frames where overlapping peaks were found at different smoothing levels, we employed a comparative algorithm to only select the peak(s) with higher prominence score. This score was obtained as the ratio of the peak’s prominence to the range of fluctuations in the correspondingly smoothed shuffle. This procedure was conducted iteratively, in increasing order of smoothing. If a broad peak overlapped with multiple narrow ones, the sum of scores of the narrow ones was compared with the broad one. To ensure that peaks at the beginning as well as the end of the movie frames were reliably detected, we circularly wrapped the movie response, for the observed as well as shuffle data.
Identifying frames with significant deviations in multiple single-unit activity (MSUA)
First, the average response across tuned neurons for each brain region was computed for each movie frame, after normalizing the response of each cell by the peak firing response. This average response was used as the observed “Multiple single unit activity (MSUA)” in Fig. 3. To compute chance level, individual neuron responses were circularly shifted with respect to the movie frames to break the frame to firing rate association but maintain overall firing rate modulation. 100 such shuffles were used, and for each shuffle, the shuffled MSUA response was computed by averaging across neurons. Across these 100 shuffles, mean and standard deviation was obtained for all frames, and used to compute the z-score of the observed MSUA. To obtain significance at p=0.025 level, Bonferroni correction was applied, and the appropriate z-score (4.04) level was chosen. The number of frames in the observed MSUA above (and below) this level were further quantified in Fig. 3. The firing deviation for these frames was computed as the ratio between the mean observed MSUA and the mean shuffled MSUA, reported as a percentage, for frames corresponding to z-score greater than +4 or less than -4. To obtain a total firing rate report, where each spike gets equal vote, we computed the total firing response by computing the total rate across all tuned neurons (and averaging by the number of neurons) in Fig. 3 and across all neurons in Extended Data Fig. 8.
Population Vector Overlap
To evaluate the properties of a population of cells, movie presentations were divided into alternate trials, yielding even and odd blocks(52). Population vector overlap was computed between the movie responses calculated separately for these 2 blocks of trials. Population vector overlap between frames x of the even trials & frame y of the odd trials was defined as the Pearson correlation coefficient between the vectors (R1,x, R2,x, … RN,x) & (R1,y, R2,y, … RN,y), where Rn,x is the mean firing rate response of the nth neuron to the xth movie frame. N is the total number of neurons used, for each brain region. This calculation was done for x and y ranging from 1 to 900, corresponding to the 900 movie frames. The same method was used for tuned and untuned neurons in continuous movie responses in Extended Data Fig. 7, and for scrambled sequence responses in Extended Data Fig. 13.
Methods similar to those previously described were used(54),(55). For tuned cells, the 60 trials of continuous movie were each decoded using all other trials. Mean firing rate responses in the 59 trials for 900 frames were used to compute a “look-up” matrix. Each neuron’s response was normalized between 0 and 1. At each frame in the “observed” trial, the correlation coefficient was computed between the population vector response in this trial and the look-up matrix. The frame corresponding to the maximal correlation was denoted as the decoded frame. Decoding error was computed as the average of the absolute difference between actual and decoded frames, across the 900 frames of the movie. For comparison, shuffle data was generated by randomly shuffling the cell-cell pairing of the look-up matrix and “observed response”. A similar procedure was used for the 20 trials of the scrambled sequence, and the corresponding middle 20 trials of the continuous movie were used.
Rearranged scrambled movie analysis
To differentiate the effects of visual content versus visual continuity between consecutive frames, we compared the responses of the same neuron to the continuous movie and the scrambled sequence. In the scrambled movie, the same visual frames as the continuous movie were used, but they were shuffled in a pseudo random fashion. The same scrambled sequence was repeated for 20 trials. The neural response was computed at each frame of the scrambled sequence, keeping the frames in the chronological order of presentation. Then the scrambled sequence of frames was rearranged to recreate the continuous movie and the corresponding neural responses computed. To address the latency between movie frame presentation and its evoked neural response, which can differ across brain regions and neurons, this calculation was repeated for rearranged scrambled sequences with variable delays between τ= -500 to +500 ms (i.e., -150 to +150 frames of 3.33ms resolution, in steps of 5 frames or 16.6ms). The correlation coefficient was computed between the continuous movie response and this variable delayed response at each delay as rmeasured (τ) = corrcoef (Rcontinuous, Rscramble-rearranged(τ)). Rcontinuous is the continuous movie response, obtained at 3.33ms resolution and similarly, Rscramble-rearranged corresponds to the scrambled response after rearrangement, at the latency τ. The latency τ yielding the largest correlation between the continuous and rearranged scrambled movie was designated as the putative response latency for that neuron. This was used in Extended Data Fig.
12. The value of rmeasured(τmax) was bootstrapped using 100 randomly generated frame reassignments, and this was used to z-score rmeasured(τmax), with z-score > 2 as criterion for significance. The resultant z-score is reported in Extended Data Fig. 12.
We thank the Allen Brain Institute for provision of the dataset, Dr. Josh Siegle for help with the dataset, Dr. Krishna Choudhary for proof-reading of the text and Dr. Massimo Scanziani for input and feedback. This work was supported by grants to M.R.M. by the National Institutes of Health NIH 1U01MH115746.
C.S.P. performed the analyses with input from M.R.M. C.S.P. and
M.R.M. wrote the manuscript.
Authors declare that they have no competing interests.
All data is publicly available at the Allen Brain Observatory – Neuropixels Visual Coding dataset (© 2019 Allen Institute, https://portal.brain-map.org/explore/circuits/visual-coding-neuropixels).
All analyses were performed using custom-written code in MATLAB version R2020a. Codes written for analysis and visualization are available from the corresponding authors at reasonable request.
Extended Data Figures
- 1.LOSS OF RECENT MEMORY AFTER BILATERAL HIPPOCAMPAL LESIONShttps://doi.org/10.1136/jnnp.20.1.11
- 2.Hippocampal “Time Cells” Bridge the Gap in Memory for Discontiguous EventsNeuron 71:737–749
- 3.Differential effects of early hippocampal pathology on episodic and semantic memoryScience (1979) 277:376–380
- 4.The hippocampus as a cognitive map
- 5.How vision and movement combine in the hippocampal place codeProceedings of the National Academy of Sciences 110:378–383
- 6.Spatial Selectivity of Rat Hippocampal Neurons: Dependence on Preparedness for MovementScience (1979) 244:1580–1582
- 7.Mapping of a non-spatial dimension by the hippocampal– entorhinal circuitNature 543:719–722
- 8.Receptive fields of single neurones in the cat’s striate cortexJ Physiol 148:574–591
- 9.The orientation and direction selectivity of cells in macaque visual cortexVision Res 22:531–544
- 10.Activity recall in a visual cortical ensembleNat Neurosci 15:449–455
- 11.A large-scale standardized physiological survey reveals functional organization of the mouse visual cortexNat Neurosci 23:138–151
- 12.Heterogeneity in the Responses of Adjacent Neurons to Natural Stimuli in Cat Striate CortexJ Neurophysiol 97:1326–1341
- 13.Natural movies evoke spike trains with low spike time variability in cat primary visual cortexJournal of Neuroscience 31:15844–15860
- 14.Experience-Dependent Asymmetric Shape of Hippocampal Receptive FieldsNeuron 25:707–715
- 15.From Hippocampus to V1: Effect of LTP on Spatio-Temporal Dynamics of Receptive FieldsNeurocomputing 32:905–911
- 16.From synaptic plasticity to spatial maps and sequence learningHippocampus 25:756–762
- 17.Experience-dependent, asymmetric expansion of hippocampal place fieldsProc Natl Acad Sci U S A 94:8918
- 18.Buzsáki, G. & Moser, E. I.Memory, navigation and theta rhythm in the hippocampal-entorhinal system. (2013) doi:10.1038/nn.3304.https://doi.org/10.1038/nn.3304
- 19.The Same Hippocampal CA1 Population Simultaneously Codes Temporal Information over Multiple TimescalesCurrent Biology 28:1499–1508
- 20.During Running in Place, Grid Cells Integrate Elapsed Time and Distance RunNeuron 88:578–589
- 21.Hippocampal ‘time cells’: time versus path integrationNeuron 78:1090–1101
- 22.Internally generated cell assembly sequences in the rat hippocampusScience (1979) 321:1322–1327
- 23.Linking hippocampal multiplexed tuning, Hebbian plasticity and navigationNature 599:442–448
- 24.Space and Time: The Hippocampus as a Sequence GeneratorTrends in Cognitive Sciences 853–869
- 25.Deciphering the hippocampal polyglot: the hippocampus as a path integration systemJ Exp Biol 199:173–85
- 26.The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving ratBrain Res 34:171–5
- 27.A selective mnemonic role for the hippocampus in monkeys: Memory for the location of objectsJournal of Neuroscience 8:4159–4167
- 28.Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C. & Fried, I.Invariant visual representation by single neurons in the human brain. 435, 1102–1107 (2005).Invariant visual representation by single neurons in the human brain 435:1102–1107
- 29.Neurons detect cognitive boundaries to structure episodic memories in humansNature Neuroscience 25:358–368
- 30.The hippocampus constructs narrative memories across distant eventsCurrent Biology 31:4935–4945
- 31.Survey of spiking in the mouse visual system reveals functional hierarchyNature 592:86–92
- 32.Dynamics and sources of response variability and its coordination in visual cortexhttps://doi.org/10.1017/S0952523819000117
- 33.Causal Influence of Visual Cues on Hippocampal Directional SelectivityCell 164:197–207
- 34.Impaired spatial selectivity and intact phase precession in two-dimensional virtual realityNat Neurosci 18:121–128
- 35.Spatial Selectivity of Unit Activity in the Hippocampal Granular LayerHIPPOCAMPUS 3:
- 36.A Quarter of a Century of Place CellsNeuron 17:813–822
- 37.Spatial correlates of firing patterns of single cells in the subiculum of the freely moving ratJournal of Neuroscience 14:2339–2356
- 38.Modulation of Visual Responses by Behavioral State in Mouse Visual CortexNeuron 65:472–479
- 39.Effects of Locomotion Extend throughout the Mouse Early Visual SystemCurrent Biology 24:2899–2907
- 40.Characterizing Speed Cells in the Rat HippocampusCell Rep 25:1872–1884
- 41.Wiener, S. I., Paul, C. A. & Eichenbaum, H.Spatial and Behavioral Correlates of Hippocampal Neuronal Activity. (1989).
- 42.O’keefe, J. & Burgess, N.Geometric determinants of the place fields of hippocampal neurons.
- 43.Multiscale representation of very large environments in the hippocampus of flying batsScience (1979) 372:
- 44.Finite Scale of Spatial Representation in the HippocampusScience (1979) 321:140–143
- 45.Dorsal CA1 hippocampal place cells form a multi-scale representation of megaspaceCurrent Biology 31:2178–2190
- 46.Large environments reveal the statistical structure governing hippocampal representationsScience (1979) 345:814–817
- 47.Fenton, A. A.et al.Behavioral/Systems/Cognitive Unmasking the CA1 Ensemble Place Code by Exposures to Small and Large Environments: More Place Cells and Multiple, Irregularly Arranged, and Expanded Place Fields in the Larger Space. (2008) doi:10.1523/JNEUROSCI.2862-08.2008.https://doi.org/10.1523/JNEUROSCI.2862-08.2008
- 48.Ensemble Place Codes in Hippocampus: CA1, CA3, and Dentate Gyrus Place Cells Have Multiple Place Fields in Large EnvironmentsPLoS One 6:22349
- 49.Functional Differences in the Backward Shifts of CA1 and CA3 Place Fields in Novel and Familiar EnvironmentsPLoS One 7:
- 50.Stable representation of a naturalistic movie emerges from episodic activity with 2 gain variabilityNature Communications 12:1–15
- 51.Synfire Chains and Cortical Songs: Temporal Modules of Cortical ActivityScience (1979) 304:559–564
- 52.The Effects of GluA1 Deletion on the Hippocampal Population Code for PositionJ Neurosci 32:8952–68
- 53.Distributed hierarchical processing in the primate cerebral cortexCereb Cortex 1:1–47
- 54.Dynamics of the hippocampal ensemble code for spaceScience (1979) 261:1055–8
- 55.Moving bar of light evokes vectorial spatial selectivity in the immobile rat hippocampus1–7https://doi.org/10.1038/s41586-022-04404-x
- 56.Disrupted Place Cell Remapping and Impaired Grid Cells in a Knockin Model of Alzheimer’s DiseaseNeuron 107:1095–1112
- 57.Grid cells without theta oscillations in the entorhinal cortex of batshttps://doi.org/10.1038/nature10583
- 58.Spatial representations of self and other in the hippocampusScience (1979) 359:213–218
- 59.Largely intact memory for spatial locations during navigation in an individual with dense amnesiaNeuropsychologia 170:108225
- 60.A Role for the Longitudinal Axis of the Hippocampus in Multiscale Representations of Large and Complex Spatial Environments and Mnemonic Hierarchieshttps://doi.org/10.5772/INTECHOPEN.71165
- 61.Robust effects of corticothalamic feedback and behavioral state on movie responses in mouse dLGNElife 11:
- 62.Multiscale temporal integration organizes hierarchical computation in human auditory cortexNat Hum Behav 6:
- 63.Temporal tuning properties along the human ventral visual streamJournal of Neuroscience 32:
- 64.A Hierarchy of Temporal Receptive Windows in Human CortexJournal of Neuroscience 28:2539–2550
- 65.Neural organization for the long-term memory ofNature 354:152–155
- 66.Expansion and Shift of Hippocampal Place Fields: Evidence for Synaptic Potentiation during Behavior741–745https://doi.org/10.1007/978-1-4757-9800-5_115
- 67.Multisensory control of hippocampal spatiotemporal selectivityScience 340:1342–6
- 68.Sequence learning and the role of the hippocampus in rodent navigation294–300https://doi.org/10.1016/j.conb.2011.12.005
- 69.Activities of visual cortical and hippocampal neurons co-fluctuate in freely moving rats during spatial behaviorElife 4:
- 70.Coherent encoding of subjective spatial position in visual cortex and hippocampusNature 562:124–127
- 71.Single-trial learning of novel stimuli by individual neurons of the human hippocampus-amygdala complexNeuron 49:805–813
- 72.Silson, E. H., Zeidman, P., Knapen, T. & Baker, C. I.Representation of contralateral visual space in the human hippocampus 1 2. (2021). doi:10.1101/2020.07.30.228361.https://doi.org/10.1101/2020.07.30.228361
- 73.The Human Brain Encodes a Chronicle of Visual Events at Each Instant of Time Through the Multiplexing of Traveling WavesJournal of Neuroscience 41:7224–7233
- 74.Nishimoto, S.et al.Reconstructing visual experiences from brain activity evoked by natural movies. 21, 1641–1646 (2011).Reconstructing visual experiences from brain activity evoked by natural movies 21:1641–1646
- 75.The hippocampus: part of an interactive posterior representational system spanning perceptual and memorial systemsJ Exp Psychol Gen 142:1242–1254
- 76.Update on Memory Systems and ProcessesNeuropsychopharmacology 36:251–273
- 77.Stringer, C.et al.Spontaneous behaviors drive multidimensional, brainwide activity. (2019) doi:10.1126/science.aav7893.https://doi.org/10.1126/science.aav7893
- 78.The Allen Mouse Brain Common Coordinate Framework: A 3D Reference Atlas ll The Allen Mouse Brain Common Coordinate Framework: A 3D Reference AtlasCell 181:936–953
- 79.Quality Metrics to Accompany Spike Sorting of Extracellular SignalsJournal of Neuroscience 31:8699–8705
- 80.Quantitative measures of cluster quality for use in extracellular recordingsNeuroscience 131:1–11