If You Flog the Data, They Will Confess

An analysis in today’s issue of The New England Journal of Medicine determined that a positive outcome in antidepressant clinical-trial results is associated with the likelihood of publication. By examining the results of 74 FDA-registered antidepressant trials (N = 12,564), the authors observed that 37 of 38 (97%) positive trials were published, but that only 14 of 36 (39%) trials determined to have questionable or negative results by the FDA were published. Moreover, among these 14 trials, 11 were published as producing positive results, in disagreement with the FDA’s questionable or negative assessment.

This analysis raises the question of data manipulation (aka “flogging”) to realize positive, statistically significant results in a clinical trial—a practice not unknown to anybody affiliated with investigative science in its academic or commercial forms. A closer look at the “questionable” or “negative” studies that were published as producing positive results indicates frequent, substantial discrepancies between the N values used in the FDA assessment and those used in the published articles, suggesting that different N values may have allowed for the calculation of statistically significant P values in the published articles. A brief glance at the published studies that are in agreement with the FDA analysis reveals few, minor, or no discrepancies in the N values used.

Otherwise, my scan of all 74 studies does not reveal any major distinctions between those in agreement with the FDA and those in disagreement; specifically, I could not find particular trends regarding journal type, antidepressant studied, or authorship affiliation (including pharmaceutical companies). A surprising number of studies in both groups indicates author affiliation with particular “Research Centers” or “Research Clinics,” which appear to be facilities that are independent of university-based medical institutions. Another quick review of the FDA’s Clinical Investigator Inspection List did not reveal that these research facilities, specifically those associated with positively spun articles, were any more likely to have cited deficiencies than those of article results in agreement with the FDA. Consequently, there appear to be few overt indicators of suspect articles that may claim dubiously spun, positive results.