Predicting perceived natural scene categories from distributed patterns of fMRI activity
Dirk Walther(1), Eamon Caddigan(1), Diane Beck(1), Li Fei-Fei(2)
(1) University of Illinois at Urbana-Champaign, (2) Princeton University
Introduction: Human observers are able to quickly and efficiently perceive the content of natural scenes (Potter, 1976). Previous studies have examined the time course of this rapid classification (Thorpe et al., 1996) as well as the brain regions activated when subjects categorize natural scenes (Epstein & Higgins, 2006). Using statistical pattern recognition algorithms similar to those employed by Cox and Savoy (2002) to decode the neural states associated with object categories, we asked whether we can identify and discriminate distributed patterns of fMRI activity associated with particular natural scene categories (beaches, mountains, forests, tall buildings, highways, and industrial scenes).
Methods: fMRI data was acquired on a 3T Siemens Allegra head scanner (gradient-echo EPI, 64 x 64 Matrix, TR = 2 s, TE = 30 ms, flip angle = 90 deg., FOV = 22 cm, 34 axial 3 mm slices without gap). Subjects viewed 100 images from each of six categories in 6 blocks of 10 images each of the same category, organized into 10 runs. A subset of the voxels most consistently activated by our images was selected via traditional univariate statistics comparing stimulus presentation versus blank screen conditions. The activation of these voxels formed the "feature set" for several pattern recognition algorithms (e.g. Support Vector Machines, Gaussian Naive Bayes). These algorithms were tested in a leave-one-run-out cross-validation procedure on the selected voxels. That is, the algorithms were trained to predict the natural scene category seen by the subject from the fMRI activity in all runs but one and tested on the left-out run. The procedure was repeated such that each run was left out once.
Results: We find that all pattern recognition algorithms predict the natural scene category seen by the subject well above chance. Furthermore, prediction accuracy was still well above chance when retinotopic cortex was excluded from the analysis, suggesting that this multi-voxel analysis does not rely solely on differences in simple visual features or differences in the retinotopic representation of the stimuli. Furthermore, we find remarkable agreement between the fMRI decoding data and behavioral data in the pattern of mis-classifications and the inversion effect for natural scenes.
Conclusions: To our knowledge, our results represent the first account of predicting the natural scene category seen by subjects from distributed fMRI activity in their brains. We address the potential confound that scenes categorization might be based on spatial frequency content as represented in retinotopic early visual areas by excluding these areas from our analysis and still obtaining well above chance prediction accuracy.
