Background: Atrial remodeling is associated with future atrial fibrillation (AF) and stroke. AF has been associated with cognitive impairment and cerebral white matter lesions. We wished to investigate the possible direct association between atrial remodeling and cerebrovascular disease in patients with and without AF documented by implantable loop recorder (ILR). Methods: Cardiac and cerebral magnetic resonance imaging were acquired in a cross-sectional study, including participants ≥70 years of age with stroke risk factors without known AF. Cerebrovascular disease was visually rated using the Fazekas scale and number of lacunar strokes. Left atrial (LA) and ventricular volumes and function were analyzed. Associations between atrial remodeling and cerebrovascular disease were assessed with logistic regression models. The analyses were stratified according to sinus rhythm or any AF during 3 months of continuous ILR monitoring to account for subclinical AF. Results: Of 200 participants investigated, 87% had a Fazekas score ≥1 and 45% had ≥1 lacunar infarct. Within 3 months of ILR monitoring, AF was detected in 28 (14%) participants. For participants with sinus rhythm (n = 172), lower LA passive emptying fraction was associated with Fazekas score after multivariable adjustment (OR [95% CI]: 0.51 [0.27; 0.86] p = 0.02), and increased LA maximum (OR [95% CI]: 1.38 [1.07; 1.82] p = 0.01) and minimum volumes (OR [95% CI]: 1.48 [1.03; 2.17] p = 0.04) were associated with lacunar infarcts. There were no significant associations in patients with AF. Conclusion: In AF-free patients, as documented by ILR monitoring, we found an independent association between LA passive emptying fraction and Fazekas score and between atrial volumes and lacunar infarcts.