This study aimed to test the hypothesis that regular physical activity is associated with improved cardiac function measured by tissue Doppler imaging (TDI) in the general population. Within a large prospective community-based population study, cardiac function was assessed in 2221 persons by TDI. Longitudinal displacement (LD), early diastolic velocity (e’), and myocardial performance index (MPI) was obtained by TDI. Linear univariable and multivariable regression analyses were performed in relation to age groups (< 50 years, 50–65 years, > 65 years) and self-reported level of physical activity: I (inactivity), II (light activity), III (moderate activity), and IV (high-level activity). Participants < 50 years in the most active group had significantly better cardiac performance when compared to all other activity levels (higher levels of e’, LD, and lower levels of MPI). The findings remained with statistical significance after adjustment for sex, ischemic heart disease, diabetes, hypertension, and body mass index (e’ = 11.0, 95% CI (10.4–11.6), p < 0.001; LD = 12.8 (12.3–13.4), p < 0.003; MPI: 0.40 (0.38–0.42), p = 0.02). In age > 65 years, there was a tendency of impaired cardiac function in higher levels of exercise. Interaction analysis revealed that age significantly modified the association between physical activity and cardiac function (p < 0.001). We found a positive association between higher level of physical activity and improved cardiac function in younger persons (< 50 years). In the general population, however, the association interacted with age and amongst persons above 65 years there was a negative association between higher level of physical activity and cardiac function.