Background: Fatigue is a major problem in multiple sclerosis (MS), and its association with MRI features is debated. Objective: To study the correlation between fatigue and lesion load, white matter (WM), and grey matter (GM), in MS patients independent of disability. Methods: We studied 222 relapsing remitting MS patients with low disability (scores ≤ 2 at the Kurtzke Expanded Disability Status Scale). Lesion load, WM and GM were measured by fully automated, operator-independent, multi-parametric segmentation method. T1 and T2 lesion volume were also measured by a semi-automated method. Fatigue was assessed by the Fatigue Severity Scale (FSS), and patients divided in high-fatigue (FSS ≥ 5; n = 197) and low-fatigue groups (FSS ≤ 4; n = 25). Results: High-fatigue patients showed significantly higher abnormal white matter fraction (AWM-f), T1 and T2 lesion loads, and significant lower WM-f, and GM-f. Multivariate analysis showed that high FSS was significantly associated with lower WM-f, and GM-f. Females and highly educated patients were significantly less fatigued. Conclusion: These results suggest that among MS patients with low disability those with high-fatigue show higher WM and GM atrophy and higher lesion load, and that female sex and higher levels of education may play a protective role towards fatigue. Furthermore, they suggest that in MS, independent of disability, WM and GM atrophy is a risk factor to have fatigue.