Peer review

From RationalWiki

Peer review is the process of subjecting scientific work to review by other experts in the field before publication. The process is designed to ensure that that the work meets the standards of the field in question and of science in general. It is obviously not foolproof, but it generally ensures that pseudoscience does not manage to masquerade as science, and that low quality work gets filtered out.

The term "peer review" is typically used for scientific and academic publications. When an article is submitted, it is sent to the authors' "peers", i.e. other experts in the same field, to assess the quality of the work.

As much effort is made as is possible to ensure that the process is "double blind", or anonymous — the original author does not know who is reviewing their work, and the reviewers are not told who the author is. However, in some fields, there are so few qualified experts that they all probably know each other, and each other's work, quite well.

[edit] Relevance to the scientific community

As practised in modern universities, peer review is a key part of the scientific method, where the goal of the system is to ensure that work is stripped of biases, assumptions, and other errors, through the scrupulous review by one's equals in the field, acting as learned professionals and mediators. Accordingly, peer in this context implies equals: i.e. the reviewers or judges should have the same or reasonably similar qualifications that the author of the work has - or claims to have. No ideology, other than the commitment to "rigorous empiricism... without which no man is a scientist," matters in asking whether a person is a scientific "peer."^[1]

Thomas Kuhn, in the process of seeking to define the nature of science, and the nature of "scientific progress," writes in his work, The Structure of Scientific Revolutions, that the unique composition of the scientific community, as composed of disinterested (read: unbiased) intellectual equals, is alone capable of generating scientific "progress."^[2] Therefore, Kuhn sees peer review as embodying "one of the strongest, if still unwritten, rules of scientific life... the prohibition of appeals to heads of state or to the populace at large in matters scientific."^[3] Simply put, politics has no place in science, and the goal of peer review is to strip politics out of the equation.

[edit] Peer review as a legal benchmark

For more on this topic, see Evidence.

Without doubt, the twentieth century has seen an uptick in the number of cases where specialized scientific knowledge is required to resolve a dispute: as a basic example, the plaintiff in a medical malpractice case must prove that a drug would have saved his father's life, before he can prove the hospital's negligence in not administering the drug. Necessarily, if a party must prove science to win his case, the court must have a way of judging the science.

Courts have met this challenge by adopting Federal Rule of Evidence 702, and the Daubert test, which require that "expert testimony" - a category which compasses scientific testimony - be methodologically sound.^[4] In law student's shorthand, the Court in a Daubert inquiry asks, before considering the scientific testimony, if the scientific theory/method relied upon by a party (1) has been peer reviewed, (2) has a low error rate, (3) is testable, and (4) is generally accepted by the scientific community. By adopting peer review as a benchmark for legal acceptance of scientific knowledge, the Daubert court validated the idea that science must be of sound, independently reviewed, unbiased methodology before it can be real science, and also agreed that the only cure for bad science is vigorous "cross examination."^[5]

The takeaway lesson is that temporal authorities, beyond the scientific community, recognize the necessity of unbiased peer review, before science can be science.

[edit] Creationists and peer review

[edit] Allegations of Bias in Peer Review

Creationists, whose "theories" have repeatedly been rejected by peer-reviewed journals,^[6] claim that the system is rigged against them, since theistic explanations for natural phenomena are rejected out of hand because of scientists' assumption of methodological naturalism when working. The argument runs something like this: institutions of peer review assume that theistic explanations are invalid, since a built-in requirement of naturalism constitutes atheistic bias.

While prima facie compelling, and probably enough to dupe a few uninformed bystanders, this argument does not hold up to logic. The purpose of science (and rationalism) is to determine explanations of natural phenomena, with reference only to objective natural indicia, so as to create a body of knowledge which all of humanity can use and apply, if possible, in engineering. The purposes of science - to determine objective truth, free of a precondition of religious belief, and to create usable knowledge - are incompatible with theistic explanations.

Of course naturalism has its limits - if no objective explanation of a phenomenon exists, perhaps some theistic agency is at work. Thus, naturalism does not displace theistic explanations; rather, it pushes back theistic explanations where they are incomplete and inaccurate. To argue that theism should be a start for science, then, completely discounts the nature of science: accusations of bias fall apart accordingly.

[edit] Attempts by creationists to intelligently design peer review

A common criticism of creationism and its intellectual kindred (baraminology, intelligent design, and creation science) is that these "sciences" can never pass peer review, as they fail to conform to scientific methodology. In addition to accusations of bias, disproven above, creationists have attempted to create their own "peer review publications," in a vain effort to create their own body of "scientific" knowledge, and render neutral the "no peer review" criticism.

[edit] First as tragedy: the creation science "mainstream"

Of course, the "symptom" that creationism lacks peer review is endemic of the "condition" that it, and its brethren, lack real scientific merit: thus, by creating new "peer review" periodicals, "cdesign proponentsists" merely cure the symptom, not the underlying condition, i.e., the unscientific nature of their theology. In short, creationism does not gain merit by virtue of having an empty label ("Now Peer Reviewed!") attached to it.

However, several creationist groups have not gotten the memo: they continue to attempt to create sham peer review journals, each of which in turn betrays the ultimate failing of their own discipline, by failing to conform to the methodology of peer review, and merely usurping the label. Several examples come to mind: Answers in Genesis, CreationWiki, JPANDS, and more "mainstream" creation scientists.

As one commentator has noted, creation "science" under its infantile peer review system approximates the level of depth of inquiry of 18th-century science.^[7] Since creation science cannot be tested without failing horribly, its intellectual discourse resembles, essentially, the creation of wild guesses, the discussion of said guesses around a fireplace, the assembly of these guesses in journals, the commitment to investigate them at some point, and then, the break for dinner. The process then repeats. The system entails no accountability because by its nature it cannot be accountable.

[edit] Then as farce: CreationWiki

For more information on CreationWiki, see its mainspace article.

CreationWiki, "a free encyclopedia of apologetics that is being assembled by an international team of missionaries,"^[8] has attempted to create a peer review system^[9] to vet CreationWiki articles, with the assumption that the result is something of "peer reviewed" quality. The assumption is either deliberate deceit or unconscious misapplication of a scientific term of art; since science depends upon a group of "peers," those being individuals versed in high sciences, not apologetics, nothing produced by CreationWiki could ever approximate peer review, unless its membership base changed by nearly 100%.

Note - parts of the following section refer to private conversations, on matters of public import. If you are the quoted private author, please comment on the Talk page to request removal of your words, but bear in mind that they are unattributed and public in scope.

Evidence suggests that the conflation of "peer review" with "casual editing" by CreationWiki is endemic of a larger confusion, within creationist circles, about the meaning of "peer review." CreationWiki administrators in particular seem to believe that "peer review" is meant to be deliberately slanted: in their own words, "that is the goal of peer reviews in general - to uphold the consensus position."^[10] Apparently, creationists reconcile their own exclusion from academic journals not by conceding their own lack of merit, but asserting that peer review, in general, is meant to "censor," and responding in kind: in their own words, "peer review is done by professionals who hold to a POV and censor other views."^[11] In short, creationists see denial of publication for lack of scientific merit as censorship for point of view: a rhetorical trick often used in creationist literature, as in Expelled, to make the cause out as a pariah.

[edit] Latecomers: Answers in Genesis

In 2008, Answers in Genesis attempted to found its own "academic" journal, Answers Research Journal.^[12] Hilarity continues to accrue at a fairly rapid rate, but it's too soon to have much to report.

[edit] References

1. ↑ See THOMAS S. KUHN, THE STRUCTURE OF SCIENTIFIC REVOLUTIONS 42 (3d ed. 1996).
2. ↑ See THOMAS S. KUHN, THE STRUCTURE OF SCIENTIFIC REVOLUTIONS 168 (3d ed. 1996).
3. ↑ Id.
4. ↑ See Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 590 (1993) (noting that the guarantee of “scientific knowledge” is one of “more than subjective belief” based in “the methods and procedures of science."
5. ↑ See id. at 597 n.13 (“This is not to say that judicial interpretation, as opposed to adjudicative fact-finding, does not share basic characteristics of the scientific endeavor.”).
6. ↑ See Intelligent design and academic freedom.
7. ↑ Hanna Rosin, God's Harvard.
8. ↑ http://creationwiki.org.
9. ↑ http://creationwiki.org/CreationWiki:Peer_review
10. ↑ Full quote:"Creationist disagree with each other on many issues. There are a variety of creationist views or opinions regarding almost every aspect of the cosmos. It is the goal of the CreationWiki to represent the main view held by creationists, and describe fringe views as such. Peer reviews are just what the phrase describes - reviews by peers. Atheists and creationists are not peers regarding theories formed from these worldviews. Only creationists can provide peer reviews of creationist views. Nevertheless, we allow for noncreationist reviews as well." CreationWiki Administrator, discussion between AmesG and CreationWiki Administrator, 10/20/2007.
11. ↑ CreationWiki Administrator, Discussion between AmesG and CreationWiki Administrator, 10/21/2007.
12. ↑ http://scienceblogs.com/pharyngula/2008/01/a_new_source_for_fake_science.php