Positron emission tomography (PET) imaging with 2-deoxy-2-[18F]-fluorodeoxyglucose (FDG) was proposed as prognostic marker in radiotherapy. Various uptake metrics and cut points were used, potentially leading to inflated effect estimates. Here, we performed a meta-analysis and systematic review of the prognostic value of pretreatment FDG-PET in head and neck squamous cell carcinoma (HNSCC) and non-small cell lung cancer (NSCLC), with tests for publication bias. Hazard ratio (HR) for overall survival (OS), disease free survival (DFS), and local control was extracted or derived from the 57 studies included. Test for publication bias was performed, and the number of statistical tests and cut-point optimizations were registered. Eggers regression related to correlation of SUVmax with OS/DFS yielded p = 0.08/p = 0.02 for HNSCC and p < 0.001/p = 0.014 for NSCLC. No outcomes showed significant correlation with SUVmax, when adjusting for publication bias effect, whereas all four showed a correlation in the conventional meta-analysis. The number of statistical tests and cut points were high with no indication of improvement over time. Our analysis showed significant evidence of publication bias leading to inflated estimates of the prognostic value of SUVmax. We suggest that improved management of these complexities, including predefined statistical analysis plans, are critical for a reliable assessment of FDG-PET.