Background: A high level of PD-L1 expression is the most relevant predictive parameter for response to immune checkpoint inhibitor (CPI) therapy in urinary bladder cancer. Existing data on the relationship between PD-L1 expression and the natural course of disease are controversial and sparse. Methods: To expand our understanding of the relationship between PD-L1 expression and parameters of cancer aggressiveness, PD-L1 was analyzed on tissue microarrays containing 2710 urothelial bladder carcinomas including 512 patients with follow-up data who underwent radical cystectomy and follow-up therapies in the pre-immune checkpoint inhibitor therapy era. Results: Tumor cell positivity in ≥10% of cells were seen in 513 (20%) and an immune cell positivity occurred in 872 (34%) of 2566 interpretable cancers. PD-L1 positivity in tumor cells increased from pTaG2 low grade (0.9% positive) to pTaG3 high grade (4.1%; p = 0.0255) and was even higher in muscle-invasive (pT2–4) carcinomas (29.3%; p < 0.0001). However, within pT2–4 carcinomas, PD-L1 positivity was linked to low pT stage (p = 0.0028), pN0 (p < 0.0001), L0 status (p = 0.0005), and a better prognosis within 512 patients with cystectomy who never received CPIs (p = 0.0073 for tumor cells and p = 0.0086 for inflammatory cells). PD-L1 staining in inflammatory cells was significantly linked to PD-L1 staining in tumor cells (p < 0.0001) and both were linked to a positive p53 immunostaining (p < 0.0001). Conclusion: It cannot be fully excluded that the strong statistical link between PD-L1 status and favorable histological tumor features as well as better prognosis could influence the outcome of studies evaluating CPIs in muscle-invasive urothelial carcinoma.