The selectivity of developmental prosopagnosia and dyslexia for stimulus category (faces or words), or process (featural or configural processing), is a subject of controversy. By manipulating featural and configural information in faces and houses, we investigated whether people with developmental prosopagnosia or dyslexia are disadvantaged in recognizing certain object classes or utilizing particular visual processing mechanisms. 34 dyslexic readers and 34 matched typical readers were tested in a delayed matching paradigm, and 28 individuals with prosopagnosia and 28 matched controls on a simultaneous matching task. Both tasks used the same stimuli. We used Representational Similarity Analysis (RSA) to correlate individual responses within each group and evaluate the similarity of these correlation matrices (reference models) with predicted data patterns (conceptual models). The reference models are correlation matrices of the accuracy of featural and configural processing of faces and houses with various difficulty levels. We created three conceptual models based on possible predicted patterns for categories, processes, and task difficulty level. If two different processes (featural or configural) or categories (faces or houses) are supported by separable mechanisms, we would expect performance for one type of process or category to better predict a trial involving an equivalent process or category than a trial involving a different process or category. RSA on behavioral data from the control groups revealed that featural and configural processes were clearly separable as were responses to stimulus category. In comparison, dyslexic readers appeared to rely on a single visual process regardless of whether features or configurations were task-relevant, and the prosopagnosia group did not perform differently based on stimulus category. We speculate that some dyslexic readers' reading deficits reflect their dependence on a single process for object recognition. In contrast, the inseparability of visual categories in individuals with prosopagnosia suggests that their recognition problem is not category selective.