Care to bet on the results of a survey of academic computer scientists? If the stakes are high enough, I could try to make it happen.

No, no more than I would bet on a survey of <insert religious group here> whether they think <religious group> is more virtuous than <non-religious group>. Academics may claim that peer review is to check validity but their actions tell a different story. This is especially true in "hard" fields like mathematics where reviewers may even struggle to follow an argument, let alone check its validity. Given that most papers are never read by others, this is really not a big deal though.

But I'll offer three further arguments for why I don't think peer review ensures validity.

Argument 1: a) Humans (including reviewers) make mistakes all the time, but b) Retractions/corrections in papers are very rare.

Unless academics are better at spotting mistakes immediately when reviewing than everyone else (they are not), we should expect lots of peer-reviewed articles to therefore have mistakes because invalid papers rarely get retracted.

Argument 2: Computer science papers don't always include reproducible software, but checking code would absolutely be required to check validity.

Argument 3: It is customary to submit papers that are rejected by one journal to another journal. This means that articles that fail "peer review" at one journal can obtain "peer review" at a different journal.

PS: For CS it's harder to check "validity", but here's how papers replicate in other fields: https://fantasticanachronism.com/2021/11/18/how-i-made-10k-predicting-which-papers-will-replicate/

Posts

Wikitag Contributions

Comments